[FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing#6248
Closed
Xeli wants to merge 26 commits intoapache:masterfrom
Closed
[FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing#6248Xeli wants to merge 26 commits intoapache:masterfrom
Xeli wants to merge 26 commits intoapache:masterfrom
Conversation
Contributor
Author
|
I've had to make a small tweak to the code ( I rebased it because I don't think anyone looked at this yet) |
Xeli
changed the title
Config example: ``` aws.credentials.provider: ASSUME_ROLE aws.credentials.provider.role.arn: <arn> aws.credentials.provider.role.sessionName: session-name aws.credentials.provider.role.provider: AUTO ``` [FLINK-9686] [kinesis] Housekeeping: Use early return instead of variable assignment and break [FLINK-9686] [kinesis] Add dependency on aws-java-sdk-sts Implicitly (via `Class.forName`) used by `STSProfileCredentialsServiceProvider`. Due to shading, it is not possible to treat this as a "provided" dependency, as Maven rewrites the class name with the shaded one, which would force clients to provide aws-java-sdk-sts shaded in the same way. [FLINK-9686] [kinesis] Mention new config option in docs [FLINK-9686] [kinesis] Use `STSAssumeRoleSessionCredentialsProvider` instead [FLINK-9686] [kinesis] Add constants for new config options This closes apache#6221.
…-pull-image option This closes apache#6232.
…tput stream This commit changes Flink such that it uses the savepoint path's file system to generate the output stream instead of the checkpoint path's file system. This closes apache#6194.
…#testRequestMemorySegmentsInterruptable()
…xception static class
…er for value with TTL This closes apache#6196. This closes apache#5799. (forgotten)
Support concurrent directory creations by accepting directories which have been created by a different thread/process in LocalFileSystem#mkdirs. This closes apache#6243.
Setting the assigned Execution#taskRestore to null after the deployment allows the JobManagerTaskRestore instance to be garbage collected. Furthermore, it won't be archived along with the Execution in the ExecutionVertex in case of a restart. This is especially important when setting state.backend.fs.memory-threshold to larger values because every state below this threshold will be stored in the meta state files and, thus, also the JobManagerTaskRestore instances. This closes apache#6251.
This closes apache#6249.
…ublic method of PubSubSource
…plement ParallelSourceFunction
Contributor
Author
|
I will close this PR for now because we'd like to add some more features such as a PubSubSink and some integrations tests |
rmetzger
added
component=Connectors/GoogleCloudPubSub
and removed
component=Connectors/Common
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Adding a PubSub connector with support for Checkpointing
Verifying this change
This change added tests and can be verified as follows:
Is there a need for integration tests? I did not see any for the other connectors.
What is a good way of testing the checkpointing / exactly-once behavior?
Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation