Skip to content

Azure: ADLSFileIO.listPrefix returns relative paths instead of fully-qualified URIs #17247

Description

@thswlsqls

Apache Iceberg version

main @ 5f3d3c5

Query engine

Spark / Flink (only on the opt-in prefix-listing path; engine-agnostic otherwise)

Please describe the bug

ADLSFileIO.listPrefix() returns each file's location as a path relative to the filesystem
container root (e.g. dir/file) instead of a fully-qualified URI — new FileInfo(pathItem.getName(), ...)
at ADLSFileIO.java:230. The siblings return full URIs: S3FileIO builds s3://bucket/key
(S3FileIO.java:348) and GCSFileIO builds gs://bucket/name (GCSFileIO.java:302).

Steps to reproduce

FileSystemWalker.listDirRecursivelyWithFileIO() (core) assumes each FileInfo.location() is a
full URI and feeds it back through ADLSFileIO. A relative path fails ADLSLocation's
URI_PATTERN (^(abfss?|wasbs?)://...), throwing ValidationException: Invalid ADLS URI: dir/file.
This is reachable from Spark DeleteOrphanFilesSparkAction with usePrefixListing(true) (opt-in,
default false) and from Flink maintenance orphan-file detection: the listed path does not match the
metadata URI, so the action fails with a BulkDeletionFailureException. No live files are deleted,
since the same validation blocks the delete.

Expected: listPrefix returns abfss://container@account.dfs.core.windows.net/dir/file, matching
S3 and GCS. The gap dates back to the initial ADLS FileIO (#8303, 2023).

Additional context

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions