[DRAFT][FLIP-597][filesystem] Add common object storage stream interfaces - #28976
Draft
Izeren wants to merge 1 commit into
Draft
[DRAFT][FLIP-597][filesystem] Add common object storage stream interfaces#28976Izeren wants to merge 1 commit into
Izeren wants to merge 1 commit into
Conversation
Introduces shared, cloud-agnostic stream abstractions in `flink-core` for building native object storage filesystem plugins without Hadoop dependencies. These types form the shared mechanics layer described in FLIP-597. This is an interfaces-only PR to gather early feedback and unblock downstream implementations. New types: - `ReadContext` / `WriteContext`: immutable opener contexts carrying read position and write metadata. - `InputStreamOpener` / `OutputStreamOpener`: cloud-agnostic functional interfaces for stream creation. - `InputStreamExtension` / `BufferingInputStreamExtension`: extension point for object-storage input streams with a default buffering implementation. - `RawAndWrappedInputStreams`: value class pairing the raw SDK stream with its wrapped counterpart. - `Abortable` (@experimental): optional capability for SDK streams that can release resources immediately from a non-owner thread, unblocking a stalled read or write. - `ObjectStorageFileSystem` (@experimental): optional filesystem capability for rich metadata access and ETag-guarded atomic rename (re-encryption). - `RichFileStatus` (@experimental): `FileStatus` extended with user-defined blob metadata and ETag. Generated-by: Claude Code
Izeren
changed the title
Izeren
changed the title
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Introduces shared, cloud-agnostic stream abstractions in
flink-corefor building native object storage filesystem plugins without Hadoop dependencies. These types form the shared mechanics layer described in FLIP-597.This is an interfaces-only draft to gather early community feedback before the concrete implementations (ObjectStorageInputStream, ObjectStorageOutputStream) are submitted.
Brief change log
ReadContext/WriteContext: immutable opener contexts carrying read position and write metadataInputStreamOpener/OutputStreamOpener: cloud-agnostic functional interfaces for stream creationInputStreamExtension/BufferingInputStreamExtension: extension point for object-storage input streams with a default buffering implementationRawAndWrappedInputStreams: value class pairing the raw SDK stream with its wrapped counterpartAbortable(@Experimental): optional capability for SDK streams that can release resources immediately from a non-owner thread, unblocking a stalled read or writeObjectStorageFileSystem(@Experimental): optional filesystem capability for rich metadata access and ETag-guarded atomic rename (re-encryption)RichFileStatus(@Experimental):FileStatusextended with user-defined blob metadata and ETagVerifying this change
This change is a trivial rework / code cleanup without any test coverage.
Concrete implementations and their tests will be submitted in a follow-up PR.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): no (all new types are@Internal/@Experimental)Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (claude-sonnet-4-6[1m])