-
Notifications
You must be signed in to change notification settings - Fork 4.5k
[BEAM-22] Remove unneccesary Fn cloning #418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The ThreadLocal-cloning Fn Factories were added in 483a5a4 but cloning was not removed (as appropriate) in that CR. |
|
The test failures are because TestDoFn does not permit itself to be reused (specifically, if There are two potential fixes. My preference would be to modify TestDoFn to be reusable, so long as the call sequence is correct (e.g. This may be a discussion suitable for the mailing list. Thoughts? |
7a830b7 to
338bc10
Compare
|
Have the tests been updated now? Should this be passing now? Also, we should state that this isn't improper cloning, but excessive cloning. It is still valid according to the model to clone these. |
The ParDoInProcessEvaluator is provided clones of a DoFn when appropriate, and should not serialize them.
338bc10 to
0cf1730
Compare
|
Updated the commit message; The RunnableOnService tests all pass, but a future-change that replaces the DirectRunner in its entirety does not, due to the implementation of |
|
LGTM but I will wait until tomorrow to merge, in case there is any last-minute feedback on BEAM-38. |
|
Based on the dev list discussion, it looks like |
Before the backoff would grow unboundedly, so we could in principle wait 1.5x to 2x the actual job time. For long running jobs this is hours. Now, we just back off at most 1 minute between checking the job state. Note there should be no danger of QPS overload here because we should have very few concurrent outstanding jobs
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Tres Seaver <tseaver@palladion.com>
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.
The ParDoInProcessEvaluator is provided clones of a DoFn when
appropriate, and should not serialize them.