Update locate_parquet_testing_files function to support hdfs input path for dataproc CI

[yinqingh]

To fix #10255

Add hdfs_glob function to support hdfs path.

[jlowe]

This should be targeted to 24.04 since the corresponding issue was moved to 24.04.

[yinqingh]

Retargeted to 24.04

[jlowe]

copyrights that I missed in the first review and a minor suggestion but otherwise lgtm.

[yinqingh]

build

[NvTimLiu]

build

[gerashegalov]

it should be possible to call FileSystem globStatus directly via PY4J without forking the JVM via hadoop fs

[yinqingh]

Hi Gera, Thanks for the review!

I tried following code to use PY4J to glob files by patterns and it works as well.
But seems it's not so straightforward like hadoop fs -ls command and also we need to process the path string further.

Considering the readability, I think probably we could keep the current implementation?

Suggested change

	cmd = ['hadoop', 'fs', '-ls', '-C', full_pattern]
	path_str = path.as_posix()
	full_pattern = path_str + '/' + pattern
	sc = get_spark_i_know_what_i_am_doing().sparkContext
	config = sc._jsc.hadoopConfiguration()
	fs_path = sc._jvm.org.apache.hadoop.fs.Path(full_pattern)
	fs = sc._jvm.org.apache.hadoop.fs.FileSystem.get(config)
	statuses = fs.globStatus(fs_path)
	for status in statuses:
	# status.getPath().toString() return string like "hdfs://hostname:8020/src/test/resources/parquet-testing/data/single_nan.parquet"
	# but pathlib.Path will remove the first "/" and convert it to "hdfs:/hostname:8020/src/test/resources/parquet-testing/data/single_nan.parquet" and then this path becomes illegal.
	# so we need to process the path like this.
	p = f'hdfs:{status.getPath().toUri().getPath()}'
	yield Path(p)

[gerashegalov]

I find it readable enough. Not a must-fix but probably would save us trouble-shooting various OOMs and other resource issues down the road. Especially in the xdist case. Keep in mind JVM initialization is slow. Hadoop adds significant XML parsing overhead on top of it. We can file it as a potential improvement and mitigation.

[yinqingh]

I will file it as a improvement. Thanks a lot!