HADOOP-19941. ABFS: Support Photon (Apache Arrow) ListBlobs on the Blob endpoint#8611
Open
bhattmanish98 wants to merge 4 commits into
Open
HADOOP-19941. ABFS: Support Photon (Apache Arrow) ListBlobs on the Blob endpoint#8611bhattmanish98 wants to merge 4 commits into
bhattmanish98 wants to merge 4 commits into
Conversation
…oint Add config-gated support for consuming ListBlobs responses in the Apache Arrow (Photon) format on the ABFS Blob endpoint, with automatic, transparent fallback to the existing XML path and no public API changes. Behaviour - New config fs.azure.photon.enabled (default false, opt-in). When enabled, ABFS advertises Arrow on ListBlobs via an Accept header (application/vnd.apache.arrow.stream, application/xml); the service may honour it or fall back to XML. - Response handling selects a parser by the response Content-Type: an Arrow content type routes to the new Arrow parser, otherwise the existing XML SAX parser is used. Both produce the same BlobListResultSchema, so all downstream processing (rename-pending filtering, pagination, directory rectification) is unchanged. - Malformed Arrow responses surface as AbfsDriverException through the existing error-handling model rather than silently degrading. Parser - ResponseParserFactory chooses between XmlListBlobResponseParser and ArrowListBlobParser behind the ListBlobResponseParser interface. - ArrowListBlobParser reads the Arrow IPC stream and reaches full parity with the XML parser: blob user metadata (Metadata map column), hdi_isfolder=true directory markers (case-insensitive; empty mkdir directories), implicit directories (BlobPrefix rows / ResourceType blobprefix|directory), copy properties, native timestamp (TimeStampSec) and unsigned length (UInt8) vectors normalized to the XML representation, and continuation via the schema NextMarker custom metadata (empty normalized to null). - The Arrow allocator memory limit is configurable, and Arrow/XML timestamps are normalized to a single RFC 1123 GMT representation so both paths yield identical FileStatus values and epochs. - Parsing is immune to thread interrupts: instead of letting ArrowStreamReader wrap the (already fully buffered) body in an interruptible NIO channel - which would abort with ClosedByInterruptException when the caller's interrupt flag is set, e.g. task cancellation - it reads through a non-interruptible channel, matching the interrupt-tolerant XML SAX path. Telemetry - Adds Photon metrics: request count, response (Arrow served) count, fallback (XML returned) count, parse-failure count, and an end-to-end listing-latency duration tracker, wired through AbfsCountersImpl and emitted from listPath. Tests - Unit tests for parser selection (TestResponseParserFactory), Arrow parsing scenarios including metadata/directory markers, blobprefix, native vector types, multi-page, special characters, allocator-limit and malformed streams, and interrupt tolerance (TestArrowListBlobParser), Photon request-header application (TestAbfsBlobClientPhotonHeaders), and metrics (TestPhotonList BlobMetrics). - Integration tests for XML/Arrow parity, fallback, pagination and metrics (ITestAbfsPhotonListStatus) plus additions to the existing list-status ITest. - Documentation updated in blobEndpoint.md; Arrow INFO logs quieted in the test log4j configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tBlobs
Address code review feedback on the Photon (Apache Arrow) ListBlobs support:
* ArrowListBlobParser: cap the per-read heap staging buffer in the
non-interruptible channel adapter at 8 KB instead of allocating a buffer
sized to the reader's full request. ArrowStreamReader reads record-batch
bodies through direct (off-heap) ByteBuffers, which take the staging path,
so unbounded requests caused large transient allocations and GC pressure
during listings. Arrow's readFully() loops on partial reads, so capping the
chunk is safe and mirrors the JDK Channels.newChannel transfer-size cap.
* ResponseParserFactory.isArrowResponse(): match the negotiated Arrow IPC
stream media type (application/vnd.apache.arrow.stream), tolerating
parameters such as charset, instead of a loose substring check for "arrow".
This prevents unrelated content types from being misrouted to the Arrow
parser. Removed the now-unused CONTENT_TYPE_ARROW_TOKEN constant and added
a regression test for content types that merely contain "arrow".
* pom.xml: remove the redundant --add-opens=java.base/java.nio=ALL-UNNAMED
from the per-profile surefire argLine entries; it is already contributed by
the shared ${maven-surefire-plugin.argLine} property.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…oton ListBlobs * ITestAzureBlobFileSystemListStatus: make the list-parse failure assertion tolerant of the XML-fallback case. The wrapping error message is derived from the parser selected by the response Content-Type, not from the Photon config, so a Photon-enabled account that returns XML yields ERR_BLOB_LIST_PARSING. Assert the message contains either the Arrow or XML parsing failure string to avoid flakiness across accounts/service versions. * DateTimeUtils: attach the caught DateTimeException as the log cause when an Arrow timestamp fails to parse, preserving the stack/details for debugging instead of only logging the raw value. * ArrowListBlobParser: reuse a single fixed-size staging buffer per non-interruptible channel instance for direct-ByteBuffer reads instead of allocating a temporary array on every read, removing avoidable allocation/GC pressure during listing. Parsing drives the channel single-threaded, so the shared buffer is safe. * AbfsHttpConstants: document that bumping ApiVersion.getCurrentVersion() to JUN_06_2026 is an intentional global default change required by the Photon (Arrow) ListBlobs contract, validated across all endpoint/operation combinations and therefore safe as the driver-wide default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
💔 -1 overall
This message was automatically generated. |
- Remove redundant lastModifiedTime()/creationTime() accessors from BlobListResultEntrySchema; use lastModified()/creation() consistently. - Fix BC_VACUOUS_INSTANCEOF in ArrowListBlobParser.readMetadata by using the typed List<?> returned by MapVector.getObject with a null check. - Replace checkstyle MagicNumber literals with named constants in DateTimeUtils and the Photon Arrow tests. - Anchor javac/java to the running JDK (java.home) in TestAggregateMetricsManager to avoid toolchain-version mismatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Adds config-gated support for consuming ListBlobs responses in the Apache Arrow (Photon) format on the ABFS Blob endpoint, with automatic, transparent fallback to the existing XML path and no public API changes.
JIRA: HADOOP-19941
Behaviour
fs.azure.photon.enabled(defaultfalse, opt-in). When enabled, ABFS advertises Arrow on ListBlobs via anAcceptheader (application/vnd.apache.arrow.stream, application/xml); the service may honour it or fall back to XML.Content-Type— Arrow routes to the new Arrow parser, otherwise the existing XML SAX parser is used. Both produce the sameBlobListResultSchema, so all downstream processing (rename-pending filtering, pagination, directory rectification) is unchanged.AbfsDriverExceptionthrough the existing error-handling model rather than silently degrading.Implementation
ResponseParserFactorychooses betweenXmlListBlobResponseParserandArrowListBlobParserbehind theListBlobResponseParserinterface.ArrowListBlobParserreaches full parity with the XML parser: blob user metadata (Metadata map column),hdi_isfolder=truedirectory markers (case-insensitive; emptymkdirdirectories), implicit directories (BlobPrefix/ResourceTypeblobprefix|directory), copy properties, nativeTimeStampSec/UInt8vectors, and continuation via schemaNextMarkermetadata.FileStatusvalues; Arrow allocator memory limit is configurable.ArrowStreamReader's interruptible NIO channel, avoidingClosedByInterruptExceptionon interrupted threads — matching the XML path.Telemetry
Adds Photon metrics: request count, response (Arrow served) count, fallback (XML returned) count, parse-failure count, and an end-to-end listing-latency duration tracker.
How was this patch tested?
blobEndpoint.md.For code changes:
org.apache.arrow:arrow-vector, Apache-2.0)