New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-19001] Add data stream api interoperability #133
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This builder is intended to be used by the DataStream embedded users, and it exposes only the RoutableMessage interface.
This commit adds the ability to construct a Sources representation from an already existing list of DataStreams that was obtained by the embedded users.
This module holds the builder classes, but it is here mainly for packaging reasons. We need to provide a jar-with-dependencies, and we can not use statefun-distribution since it also contains the connectors that users don't need.
flink-streaming-java should always be with scope provided. It wasn't explicitly needed before, because this artifact was not part of any jar-with-dependencies.
Since the move to 1.11, Flink executors are refactored out of flink-streaming-java and moved into flink-clients which is a required dependency now.
This commit serves multiple purposes: 1) Since the providers are initalized also during planning, making the http client lazily initalized would improve planning time. 2) It would allow making the provider serializable.
FunctionType is a very simple value type, that is now also used by the data-stream api bridge (requires java serialization)
This commit makes the internal class HttpFunctionSpec java serializable, so that it can be used from the statefun-datastream-api. If at any point we will decide to add a non (trivally serializable) member to this class, then we can undo this change and provide a serializable "twin" at the datastream api.
This commits adds a serializable http function provider that delegates to the real http function provider at runtime.
This commits add a request reply remote function builder, to be used as a public API. This class delegates its methods to the non-public (internal) HttpSpec.Builder class.
#docs |
Would be followed on the docs sprint, thanks for the reminder. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add DataStream API interoperability
This PR adds an extension the stateful function SDK that allows embedding stateful function applications into a data stream program.
This integration allows:
DataStream
s as StateFun ingressesRequestReply
functionsDataStream
s.Here is a short example snippet of how to insert a StateFun pipeline into a regular DataStream pipeline. (full example)
RoutableMessage
is the entry point to the StateFun pipeline.DataStream<RoutableMessage>
as ingresses.payloads
associated with theRoutableMessage
to the appropriate stateful function instance.DataStream
fromStatefulFunctionEgressStreams
.The following is a short description of the changes (not in the same commit order)
statefun-flink/statefun-flink-datastream
for the new SDK.