Checking for job process status inside a map task attempt when it's not ... #126
Conversation
…ot the first attempt
|
The idea behind this pull request is to optimize the processing time when attempts fail and are restarted. In such cases, a previous attempt has already run through the process record and failed at a later point in processing. The next attempt that is brought up will now reprocess all the records that the previous (failed) attempt has already processed. This results in a lot of time spent in this hRaven job and the next hRaven run cant start up. |
| Get get = new Get(row); | ||
| get.addColumn(Constants.INFO_FAM_BYTES, Constants.JOB_PROCESSED_SUCCESS_COL_BYTES); | ||
|
|
||
| boolean success = false; |
There was a problem hiding this comment.
Nit: might be bit easier to read the code it this is named something like "processed" or "alreadyProcessed". "Success" seems a little confusing.
| } | ||
| } | ||
| return processed; | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
Probably best to capture only the kinds of exceptions that can be thrown by the code you run.
Let's make sure that we don't catch other exceptions such as InterruptedException, or OOME's.
…ocessing_in_mapper Checking for job process status inside a map task attempt when it's not ...
...the first attempt