HBASE-23601: OutputSink.WriterThread exception gets stuck and repeated indefinietly #956
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d indefinietly clear exception after logged try to restart writer threads if needed
wchevreuil
reviewed
Dec 20, 2019
|
🎊 +1 overall
This message was automatically generated. |
saintstack
reviewed
Dec 20, 2019
| for (WriterThread t : writerThreads) { | ||
| if (!t.isAlive()){ | ||
| LOG.debug("restarting thread" + t); | ||
| t.start(); |
There was a problem hiding this comment.
I didn't think you could restart a dead thread. You have to create a new one?
There was a problem hiding this comment.
You are absolutely. Thanks for pointing it out.
|
|
||
| public Throwable clearError(){ | ||
| return thrown.getAndSet(null); | ||
| } |
There was a problem hiding this comment.
I see we make the pipelinecontroller once and use it thereafter -- it goes down into the WriterThread.... . I was going to suggest just create a new Controller on each write but looks like too much of a refactor.
Would it be cleaner if checkForErrors cleared any errors found... ; i.e. checkForErrors throws but before it throws it clears any exception in Controller?
There was a problem hiding this comment.
Not just cleaner it would also cover more usecases. I moved that line into the checkForErrors.
HorizonNet
reviewed
Dec 30, 2019
…d indefinietly remove dead writer threads and create new ones reusing the old names remove the stored exception before re-throwing it add unit test
…d indefinietly add missing changes from prev commit
|
@wchevreuil > Is it possible to add a test that reproduces the problem? @HorizonNet > Does this problem only occur in HBase 2.2.x? If not it would be better to target the master branch. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
reviewed
Jan 3, 2020
| while(writerIterator.hasNext()){ | ||
| WriterThread t = writerIterator.next(); | ||
| if (!t.isAlive()){ | ||
| names.add(t.getName()); |
There was a problem hiding this comment.
Would it be easier to move through the writerThread array using an index replacing dead threads with new live ones as you go? Should log when you replace a thread I'd say at DEBUG level?
There was a problem hiding this comment.
Thanks for the suggestion. Added it to the next commit.
| this.entryBuffers = entryBuffers; | ||
| outputSink = sink; | ||
| } | ||
|
|
There was a problem hiding this comment.
Have the constructor just above this call this one passing the constructed thread name? nit.
…d indefinietly fix checkstyle validations add license to test class code improvements
|
🎊 +1 overall
This message was automatically generated. |
saintstack
approved these changes
Jan 9, 2020
asfgit
pushed a commit
that referenced
this pull request
Jan 9, 2020
…d indefinietly (#956) * HBASE-23601: OutputSink.WriterThread exception gets stuck and repeated indefinietly clear exception after logged try to restart writer threads if needed
asfgit
pushed a commit
that referenced
this pull request
Jan 9, 2020
…d indefinietly (#956) * HBASE-23601: OutputSink.WriterThread exception gets stuck and repeated indefinietly clear exception after logged try to restart writer threads if needed
BukrosSzabolcs
added a commit
to BukrosSzabolcs/hbase
that referenced
this pull request
Jan 13, 2020
…d indefinietly (apache#956) * HBASE-23601: OutputSink.WriterThread exception gets stuck and repeated indefinietly clear exception after logged try to restart writer threads if needed
|
created a new PR for branch-2 #1028 |
symat
pushed a commit
to symat/hbase
that referenced
this pull request
Feb 17, 2021
…d indefinietly (apache#956) * HBASE-23601: OutputSink.WriterThread exception gets stuck and repeated indefinietly clear exception after logged try to restart writer threads if needed (cherry picked from commit ce7c559) Change-Id: I62ede7ebc02213357678e50b9b995b00dfac4a5d
symat
pushed a commit
to symat/hbase
that referenced
this pull request
Feb 17, 2021
… repeated indefinietly (apache#956)" This reverts commit ce7c559. (cherry picked from commit fd6eb38) Change-Id: I67ab28228bec55a17d2b8c01946be264723ad2a6
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
clear exception after logged
try to restart writer threads if needed