-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-2918] Add state support for batch in portable FlinkRunner #6740
Conversation
64c0072
to
889b3f2
Compare
Run Python Flink ValidatesRunner |
Run Java Flink PortableValidatesRunner |
There is still an issue I'm working on. Will ping when this is ready for review. |
889b3f2
to
31ea165
Compare
Run Python Flink ValidatesRunner |
Run Java Flink PortableValidatesRunner |
This is now ready for review. |
31ea165
to
896da6c
Compare
Rebased to #6726 and enabled additional |
Run Python Flink ValidatesRunner |
Run Java Flink PortableValidatesRunner |
896da6c
to
e152b6a
Compare
Rebased to master after #6726 was merged. |
Run Python Flink ValidatesRunner |
Run Java Flink PortableValidatesRunner |
@tweise Do you think you could take a look? |
@@ -595,4 +624,15 @@ private static void pruneOutput( | |||
String.format("%s/out.%d", transformName, unionTag)); | |||
context.addDataSet(collectionId, pruningOperator); | |||
} | |||
|
|||
private static <T> Coder<WindowedValue<T>> instantiateCoder( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to FlinkPipelineTranslatorUtils
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved.
Looks good, just one minor suggestion. |
29e1e1a
to
d558367
Compare
This adds portable state support for the batch mode.
MapState and SetState are not yet supported by the Fn client and need to be disabled.
d558367
to
d448a96
Compare
This adds portable state support for the batch mode.
CC @tweise @angoenka @robertwb
Post-Commit Tests Status (on master branch)