Apache Iceberg version
main @ 5f3d3c5
Query engine
Spark / Flink (only on the opt-in prefix-listing path; engine-agnostic otherwise)
Please describe the bug
ADLSFileIO.listPrefix() returns each file's location as a path relative to the filesystem
container root (e.g. dir/file) instead of a fully-qualified URI — new FileInfo(pathItem.getName(), ...)
at ADLSFileIO.java:230. The siblings return full URIs: S3FileIO builds s3://bucket/key
(S3FileIO.java:348) and GCSFileIO builds gs://bucket/name (GCSFileIO.java:302).
Steps to reproduce
FileSystemWalker.listDirRecursivelyWithFileIO() (core) assumes each FileInfo.location() is a
full URI and feeds it back through ADLSFileIO. A relative path fails ADLSLocation's
URI_PATTERN (^(abfss?|wasbs?)://...), throwing ValidationException: Invalid ADLS URI: dir/file.
This is reachable from Spark DeleteOrphanFilesSparkAction with usePrefixListing(true) (opt-in,
default false) and from Flink maintenance orphan-file detection: the listed path does not match the
metadata URI, so the action fails with a BulkDeletionFailureException. No live files are deleted,
since the same validation blocks the delete.
Expected: listPrefix returns abfss://container@account.dfs.core.windows.net/dir/file, matching
S3 and GCS. The gap dates back to the initial ADLS FileIO (#8303, 2023).
Additional context
N/A
Apache Iceberg version
main @ 5f3d3c5
Query engine
Spark / Flink (only on the opt-in prefix-listing path; engine-agnostic otherwise)
Please describe the bug
ADLSFileIO.listPrefix()returns each file's location as a path relative to the filesystemcontainer root (e.g.
dir/file) instead of a fully-qualified URI —new FileInfo(pathItem.getName(), ...)at ADLSFileIO.java:230. The siblings return full URIs:
S3FileIObuildss3://bucket/key(S3FileIO.java:348) and
GCSFileIObuildsgs://bucket/name(GCSFileIO.java:302).Steps to reproduce
FileSystemWalker.listDirRecursivelyWithFileIO()(core) assumes eachFileInfo.location()is afull URI and feeds it back through
ADLSFileIO. A relative path failsADLSLocation'sURI_PATTERN(^(abfss?|wasbs?)://...), throwingValidationException: Invalid ADLS URI: dir/file.This is reachable from Spark
DeleteOrphanFilesSparkActionwithusePrefixListing(true)(opt-in,default false) and from Flink maintenance orphan-file detection: the listed path does not match the
metadata URI, so the action fails with a
BulkDeletionFailureException. No live files are deleted,since the same validation blocks the delete.
Expected:
listPrefixreturnsabfss://container@account.dfs.core.windows.net/dir/file, matchingS3 and GCS. The gap dates back to the initial ADLS FileIO (#8303, 2023).
Additional context
N/A