-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-11042] Remove DirectRunner dependency on deprecated CreatePCollectionView primitive #13043
[BEAM-11042] Remove DirectRunner dependency on deprecated CreatePCollectionView primitive #13043
Conversation
As it turns out, I tested this on a branch where I had removed |
fa8231d
to
9c97490
Compare
This is fun. If the rewrites are in one order, just |
45ac573
to
bb30001
Compare
b7b51e4
to
de34ae4
Compare
Fixed up the issue. It was the heavily mocked out direct runner tests that needed to be adjusted. Please take a look. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes. Overall looks good to me except one minor comment.
import org.apache.beam.sdk.util.WindowedValue; | ||
import org.checkerframework.checker.nullness.qual.Nullable; | ||
|
||
class CreateViewNoopEvaluatorFactory implements TransformEvaluatorFactory { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add some comments here to explain why we need a no-op here? Also can we attach a TODO here to clean it up in the future?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a big comment and BEAM-11049.
de34ae4
to
e2b129f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Because you approved with one comment, I already fixed the commit history and force pushed to the PR branch. I will wait for some green to make sure I didn't accidentally insert an error when editing the comments, then merge.
import org.apache.beam.sdk.util.WindowedValue; | ||
import org.checkerframework.checker.nullness.qual.Nullable; | ||
|
||
class CreateViewNoopEvaluatorFactory implements TransformEvaluatorFactory { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a big comment and BEAM-11049.
Run Java PreCommit |
Run Java PreCommit (flake in attempted metrics) |
Run Java PreCommit |
Same failure. Since it was green before and I only changed comments, I still believe it is a flake. I wonder if this PR changed some flake rate. |
I built this change on top of #13042 and confirmed that the direct runner's
ValidatesRunner
still passed. I confirmed withgrep -r CreatePCollectionView runners/direct-java
that there are no more mentions in the direct runner.Summary of how the change works:
ParDo.MultiOutput
primitives, attach aWriteView
direct runner step to each side input. This has the same behavior as today but does not depend on having aCreatePCollectionView
primitive in the SDK graph.CreatePCollectionView
is still in the graph and it is not that easy to delete, just attach a noop evaluation. WhenCreatePCollectionView
is deleted soon, this can be removed.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username
).[BEAM-XXX] Fixes bug in ApproximateQuantiles
, where you replaceBEAM-XXX
with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.