pyarrow.fs.FileSystem.from_uri does not refer to HDFS core-site.xml config file when resolving namenode for HDFS URL

[wkarwacki]

pyarrow==15.0.2

Hey! I'm trying to use this function https://arrow.apache.org/docs/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem.from_uri to obtain FileSystem like below:

from pyarrow.fs import FileSystem

FileSystem.from_uri("hdfs:///some/path")

however, even though core-site.xml is properly configured I'm getting:

URISyntaxException: Expected authority at index 7: hdfs://java.lang.IllegalArgumentException: Expected authority at index 7: hdfs://

I might be mistaken, however it seems that it tries to find hostname between second and third slash charater in such HDFS URL and just does not take into account core-site.xml config at all.

I'm able to successfully create FileSystem when providing namenode explicitly with hdfs://{namenode}/{path}. Currently, we are working around this issue with two strategies of parsing:

from pyarrow.fs import FileSystem, HadoopFileSystem

parsed = urlparse(url)
if parsed.scheme == "hdfs":
    hadoop_file_system = HadoopFileSystem("default") # this properly recognizes core-site.xml
    return (hadoop_file_system, parsed.path)
else:
    file_system: Tuple[FileSystem, str] = FileSystem.from_uri(uri=url)
    return file_system

Component(s)

C++, Python

[pstrzelczak]

@pitrou @kszucs @westonpace @kou could you check if this can be fixed easily?

With this issue, using some pyarrow API requires either opening the file upfront and passing file like object (e.g. read_csv or open_csv) or passing filesystem parameter explicitly:

>>> import pyarrow.parquet as pq
>>> pq.read_metadata('hdfs:///foo.parquet')
log4j:WARN No appenders could be found for logger (org.apache.hadoop.fs.FileSystem).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
hdfsBuilderConnect(forceNewInstance=1, nn=hdfs://, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
URISyntaxException: Expected authority at index 7: hdfs://java.lang.IllegalArgumentException: Expected authority at index 7: hdfs://
	at java.base/java.net.URI.create(URI.java:906)
Caused by: java.net.URISyntaxException: Expected authority at index 7: hdfs://
	at java.base/java.net.URI$Parser.fail(URI.java:2976)
	at java.base/java.net.URI$Parser.failExpecting(URI.java:2982)
	at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3226)
	at java.base/java.net.URI$Parser.parse(URI.java:3177)
	at java.base/java.net.URI.<init>(URI.java:623)
	at java.base/java.net.URI.create(URI.java:904)
/arrow/cpp/src/arrow/status.cc:168: Failed to disconnect hdfs client: IOError: HDFS hdfsFS::Disconnect failed. Detail: [errno 9] Bad file descriptor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2341, in read_metadata
    filesystem, where = _resolve_filesystem_and_path(where, filesystem)
  File "/usr/local/lib/python3.10/site-packages/pyarrow/fs.py", line 174, in _resolve_filesystem_and_path
    filesystem, path = FileSystem.from_uri(path)
  File "pyarrow/_fs.pyx", line 502, in pyarrow._fs.FileSystem.from_uri
  File "pyarrow/_fs.pyx", line 457, in pyarrow._fs.FileSystem._native_from_uri
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: HDFS connection failed
>>> import fsspec
>>> pq.read_metadata('hdfs:///foo.parquet', filesystem=fsspec.filesystem('hdfs'))
<pyarrow._parquet.FileMetaData object at 0x798066b93bf0>
  created_by: parquet-cpp-arrow version 21.0.0
  num_columns: 6
  num_rows: 150
  num_row_groups: 1
  format_version: 2.6
  serialized_size: 1418

[kszucs]

@pstrzelczak can you try to use "default" as the hostname component of the URI? According to #47560 it should be handled by libhdfs.

[pstrzelczak]

@kszucs I want to avoid modifying valid HDFS URIs.