[SPARK-22955][DSTREAMS] - graceful shutdown shouldn't lead to job gen… #25511
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
choojoyq
force-pushed
the
SPARK-22955-job-generation-error-on-graceful-stop
branch
from
2ba6586 to
c8937b1
Compare
Author
|
Hi @srowen, could you please take a look ? |
Member
|
I tend to agree, but let's see what tests say. CC @tdas |
|
Test build #4840 has finished for PR 25511 at commit
|
Member
|
Merged to master |
Contributor
Member
|
Yes I think that's OK. We're in a 'code freeze' for the 2.4.4 release at the moment, so I hesitate to merge anything but critical fixes until it's finalized. But it could go in for 2.4.5. |
Contributor
|
OK, thanks. |
Contributor
Member
|
@dongjoon-hyun did you say you're having problems with the merge script not being able to backport? I'm seeing the same. We are just manually cherry-picking and pushing? |
srowen
pushed a commit
that referenced
this pull request
Sep 3, 2019
### What changes were proposed in this pull request? During graceful shutdown of ``StreamingContext`` ``graph.stop()`` is invoked right after stopping of ``timer`` which generates new job. Thus it's possible that the latest jobs generated by timer are still in the middle of generation but invocation of ``graph.stop()`` closes some objects required to job generation, e.g. consumer for Kafka, and generation fails. That also leads to fully waiting of ``spark.streaming.gracefulStopTimeout`` which is equal to 10 batch intervals by default. Stopping of the graph should be performed later, after ``haveAllBatchesBeenProcessed`` is completed. ### How was this patch tested? Added test to existing test suite. Closes #25511 from choojoyq/SPARK-22955-job-generation-error-on-graceful-stop. Authored-by: Nikita Gorbachevsky <nikitag@playtika.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
Member
|
Seemed to work fine. This is backported to 2.4 |
Contributor
|
Thank @srowen |
Member
|
Yes. @srowen . The current merge script has two issues.
BTW, sorry for being late response, I've been on a vacation since last Saturday. I'm connecting here time to time. |
rluta
pushed a commit
to rluta/spark
that referenced
this pull request
Sep 17, 2019
### What changes were proposed in this pull request? During graceful shutdown of ``StreamingContext`` ``graph.stop()`` is invoked right after stopping of ``timer`` which generates new job. Thus it's possible that the latest jobs generated by timer are still in the middle of generation but invocation of ``graph.stop()`` closes some objects required to job generation, e.g. consumer for Kafka, and generation fails. That also leads to fully waiting of ``spark.streaming.gracefulStopTimeout`` which is equal to 10 batch intervals by default. Stopping of the graph should be performed later, after ``haveAllBatchesBeenProcessed`` is completed. ### How was this patch tested? Added test to existing test suite. Closes apache#25511 from choojoyq/SPARK-22955-job-generation-error-on-graceful-stop. Authored-by: Nikita Gorbachevsky <nikitag@playtika.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
During graceful shutdown of
StreamingContextgraph.stop()is invoked right after stopping oftimerwhich generates new job. Thus it's possible that the latest jobs generated by timer are still in the middle of generation but invocation ofgraph.stop()closes some objects required to job generation, e.g. consumer for Kafka, and generation fails. That also leads to fully waiting ofspark.streaming.gracefulStopTimeoutwhich is equal to 10 batch intervals by default. Stopping of the graph should be performed later, afterhaveAllBatchesBeenProcessedis completed.How was this patch tested?
Added test to existing test suite.