HIVE-24068: Add re-execution plugin for handling DAG submission and unmanaged AM failures #1428
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kgyrtkirk
reviewed
Aug 25, 2020
There was a problem hiding this comment.
there were some other strange things...and it turns out that HIVE-23725 have messed up things around here....this patch at least follows the existing concepts....
+1
Comment on lines
44
to
46
| // there could be race condition where getSession could return a healthy AM but by the time DAG is submitted | ||
| // the AM could become unhealthy/unreachable (possible DNS or network issues) which can fail tez DAG | ||
| // submission. Since the DAG hasn't started execution yet this failure can be safely restarted. |
There was a problem hiding this comment.
I think this explanation could be moved into the class documentation
…o lost AM query plugin.
prasanthj
changed the title
|
@kgyrtkirk thanks for the review! Addressed the review comment. Also handled the maxExecutions within the plugin which was missing before. Once minor change added is to update AM loss plugin to handle unmanaged AM failure as well. |
kgyrtkirk
added
tests failed
tests pending
tests passed
and removed
tests pending
tests failed
labels
saihemanth-cloudera
pushed a commit
to saihemanth-cloudera/hive
that referenced
this pull request
Sep 1, 2020
…nmanaged AM failures (apache#1428) * HIVE-24068: Add re-execution plugin for handling DAG submission failures * addressed Zoltan's code review comments. Added unmanaged AM failure to lost AM query plugin. * fix comments Co-authored-by: Prasanth Jayachandran <pjayachandran@cloudera.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DAG submission failure can also happen in environments where AM container died causing DNS issues. DAG submissions are safe to retry as the DAG hasn't started execution yet. There are retries at getSession and submitDAG level individually but some submitDAG failure has to retry getSession as well as AM could be unreachable, this can be handled in re-execution plugin. This PR adds a new re-execution plugin for intermittent DAG submission failures.
Why are the changes needed?
To make hive resilient to environments with network/DNS issues.
Does this PR introduce any user-facing change?
Yes. Adds the re-exec plugin as default option.
How was this patch tested?
Manually. Tez code was changed to explicitly throw UnknownHostException to simulate DNS/network issue and tested to make sure retry happens.