[BEAM-7412] shut down executor service in fn harness #8722
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iemejia
reviewed
May 30, 2019
| @@ -203,7 +204,9 @@ public static void main( | |||
| JvmInitializers.runBeforeProcessing(options); | |||
|
|
|||
| LOG.info("Entering instruction processing loop"); | |||
| control.processInstructionRequests(options.as(GcsOptions.class).getExecutorService()); | |||
| ExecutorService executorService = options.as(GcsOptions.class).getExecutorService(); | |||
There was a problem hiding this comment.
This does not come from this PR, but shouldn't this be in a more general options type than GcsOptions ? I mean SparkPipelineOptions does not have this thing, so not sure if this will have an effect.
There was a problem hiding this comment.
Also shouldn't a change in the harness trigger other tests to validate that we don't break portability (Flink PVR for example). (Note also unrelated to this PR, just thinking on things that can be done).
There was a problem hiding this comment.
agreed on both points. will have to run spark and flink pvr tests manually.
|
Run Java Flink PortableValidatesRunner Batch |
|
Run Java Flink PortableValidatesRunner Batch |
angoenka
requested changes
May 30, 2019
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
Outdated
Show resolved
Hide resolved
|
Run JavaPortabilityApi PreCommit |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was somewhat surprised to find that the unbounded thread growth I encountered when running portable validates runner tests on local Spark is was actually caused by
GcsOptions.java. I'm guessing this hasn't caused problems before because the SDK harness was always run in a separately managed process/container/whatever that terminated when work was complete.Incidentally, this was a pain to track down because thousands of threads used for any number of purposes were all uniformly named
pool-N-thread-M(as they all used Java'sdefaultThreadFactory). Going forward, I might look into naming Beam threads so it's easier to debug and analyze performance.R: @angoenka
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.