HBASE-22422 Retain an ByteBuff with refCnt=0 when getBlock from LRUCache #242
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Apache9
requested changes
May 16, 2019
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
Show resolved
Hide resolved
| // Decrease the block's reference count, and if refCount is 0, then it'll auto-deallocate. DO | ||
| // NOT move this up because if do that then the victimHandler may access the buffer with | ||
| // refCnt = 0 which is disallowed. | ||
| previous.getBuffer().release(); |
Contributor
There was a problem hiding this comment.
So this is the problem. Mind explaining more? Why in victimHandler we will access the previous? And is it possible to add a UT?
Member
Author
There was a problem hiding this comment.
Why in victimHandler we will access the previous? And is it possible to add a UT?
For InclusiveCombinedBlockCache , we will move the evicted block from LRUCache to an larger L2 cache (such as MemcachedBlockCache for longer caching I think), So if release in line#596, then victimHandler will cache an block which point to an unknown area because its memory has been free.
Yeah, will provide a UT.
|
💔 -1 overall
This message was automatically generated. |
Member
Author
|
The patch still not fix the refCnt=0 retain issue, because I've applied this patch to my local branch , and deploy to my test cluster. After run some days, the QPS still dropped from 300000 Get/second to 200 Get/second. Need to find out why. |
Contributor
|
So even after this patch if QPS drops - you stlil get the exception as attached in the original JIRA description? |
Member
Author
|
bq. So even after this patch if QPS drops - you stlil get the exception as attached in the original JIRA description? |
Member
Author
|
Update the patch with the final fix and UT, FYI @Apache9 @ramkrish86 @anoopsjohn |
Member
Author
|
Seems no hadoop QA feedback ? It's strange... |
ramkrish86
reviewed
May 27, 2019
| // Must initialize it with null here, because if don't and once an exception happen in | ||
| // readBlock, then we'll release the previous assigned block twice in the finally block. | ||
| // (See HBASE-22422) | ||
| block = null; |
Apache9
approved these changes
May 27, 2019
| BlockCacheKey cacheKey, Cacheable newBlock) { | ||
| // NOTICE: The getBlock has retained the existingBlock inside. | ||
| Cacheable existingBlock = blockCache.getBlock(cacheKey, false, false, false); | ||
| if (existingBlock == null) { |
Contributor
There was a problem hiding this comment.
Oh, this means we may get NPE in the past?
Member
Author
There was a problem hiding this comment.
Yes, see the LruBlockCache#cacheBlock:
LruCachedBlock cb = map.get(cacheKey);
if (cb != null && !BlockCacheUtil.shouldReplaceExistingCacheBlock(this, cacheKey, buf)) {
return;
}
The existence pre-check and accessing block in shouldReplaceExistingCacheBlock is not atomic op. so if any eviction happen between them, the NPE will happen. I added a UT testMultiThreadGetAndEvictBlock to address this.
ramkrish86
reviewed
May 29, 2019
| return absent.get() ? null : re; | ||
| } | ||
|
|
||
| public boolean remove(BlockCacheKey key) { |
Contributor
There was a problem hiding this comment.
While removing the atomicity is not needed? Because we do computeIfAbsent and that is already now atomically guarded?
ramkrish86
reviewed
May 29, 2019
| absent.set(true); | ||
| return entry; | ||
| }); | ||
| return absent.get() ? null : re; |
Contributor
There was a problem hiding this comment.
I think I got it now. You have changed the get() to computeIfPresent(). So if the remove has already removed the entry then the get() cannot do a retain(). So a similar change is also needed for putIfAbsent() also? Rest looks good to me.
Member
Author
There was a problem hiding this comment.
Yeah, if don't make the put and retain in atomic , then if remove & release happen between the put and retain, finally we 're retain a block with refCnt=0 which is also disallowed. Thanks.
asfgit
pushed a commit
that referenced
this pull request
May 31, 2019
asfgit
pushed a commit
that referenced
this pull request
Jun 18, 2019
asfgit
pushed a commit
that referenced
this pull request
Jun 24, 2019
openinx
added a commit
to openinx/hbase
that referenced
this pull request
Jun 25, 2019
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.