[FLINK-11838] please ignore this PR #18275
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add new project for GS file system and recoverable writer. This includes FileSystemFactory SPI wireup, but with no recoverable writer (yet).
Add interfaces to abstract away direct access to the Google Storage API, both to simplify that access and to make it mockable.
…orage Add GSBlobStorage, an implementation of BlobStorage against the Google Storage API.
Add some utility functions used by the recoverable writer. Includes unit tests.
Add MockBlobStorage, an implementation of BlobStorage against an in-memory store.
Implements RecoverableWriter against the BlobStorage abstraction and wires up HadoopFileSystem.createRecoverableWriter for GS. Includes unit tests.
Rename BlobStorage and GSBlobStorage to GSBlobStorage and GSBlobStorageImpl Rename BlobStorage.write to BlobStorage#writeBlob Remove ability to specify blob content type Introduce GSBlobIdentifier abstraction Remove BlobId normalization (no longer needed) Move nested interfaces/classes to bottom Add count<=32 precondition for GSBlobStorageImpl#compose Other code improvements
…base classes for new scheme/project Remove unnecessary config options Better handle optional config options Change WRITER_CHUNK_SIZE to use .memoryType() instead of .intType() and add check that value is multiple of 256KB Make HadoopFileSystem and FileSystem aware of gs: scheme and flink-gs-fs-hadoop project
…ements Refactor utilities Validate gs scheme in parseUri Remove GSRecoverableWriteState, replace with immutable GSResumeRecoverable and private state in GSRecoverableFsDataOutputStream Fix commit, commitAfterRecovery, cleanupRecoverableState, and requiresCleanupOfRecoverableState to implement proper commit and cleanup algorithms
Exclude javax.annotation-api artifact because of its GPLv2 license, which cannot be included in Flink artifacts. Everything seems to build and run fine without it.
…w feedback Remove unnecessary throw clauses Inline call to preconditions.checkNotNull Use unmodifiable list for component object ids in GSResumeRecoverable Overload GSBlobStorage#writeBlob to allow setting chunk size, remove setChunkSize on GSBlobStorage#WriteChannel
…ing separating commit vs. resume recoverables Separate commit vs. resume recoverables Refactor utility functions Add javadoc comments
|
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit a659f92 (Wed Jan 05 20:27:53 UTC 2022) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
|
CI Run with DL logs: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28990&view=results |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is for testing issues with CI