HDFS-17953. DataNode IBR pendingIBRs grows unbounded when a NameNode is unreachable, causing OOM#8626
Open
singer-bin wants to merge 1 commit into
Open
HDFS-17953. DataNode IBR pendingIBRs grows unbounded when a NameNode is unreachable, causing OOM#8626singer-bin wants to merge 1 commit into
singer-bin wants to merge 1 commit into
Conversation
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
@singer-bin Great catch! I am confused why we need |
…is unreachable, causing OOM
Contributor
Author
|
Thanks for the review, @Hexiaoqiao ! You are right — the pending-block-count cap alone is enough to bound the DataNode heap footprint, so The fix now keeps a single, simple guard: Updated tests (all passing locally):
Please take another look. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: HDFS-17953
Problem
When a DataNode is configured with HA NameNodes (
dfs.ha.namenodes.<nsId>), butone of the configured NameNode IDs has no (or a wrong) RPC address — or the
NameNode is permanently unreachable — the
IncrementalBlockReportManager.pendingIBRsmap for that actor grows withoutbound and eventually causes the DataNode to OOM.
In
BPOfferService#notifyNamenodeBlock, every block receive/delete event isadded to the IBR queue of every
BPServiceActor, including the actor for theunreachable NameNode. In
IncrementalBlockReportManager#sendIBRs, on sendfailure
putMissing()puts all blocks back into the queue. When the NameNode ispermanently unreachable the send always fails, new events keep arriving, and the
queue grows indefinitely — there is no size limit, no TTL and no eviction.
The only existing protection (HDFS-9917) calls
clearIBRs()duringreRegister()for STANDBY/OBSERVER NNs, butreRegister()is never triggeredwhen the NN is completely unreachable (no
DNA_REGISTERcommand can bereceived). A production heap dump showed ~223M
ReceivedDeletedBlockInfoinstances occupying ~30 GB, caused by a stale
nn3whose RPC address was neverconfigured (it fell back to the logical
nameservice:8020which neverresolves).
Fix
dfs.datanode.ibr.max.pending.size(default1,000,000). When exceeded, the pending IBR queue is cleared with a warning.
dfs.datanode.ibr.max.stale.interval.ms(default30 min): if no successful IBR send happens within this window while entries
are pending, the queue is cleared.
the NameNode gets a complete, consistent view once reachable again.
This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
optimization for incremental updates between FBRs. Dropping queued IBRs for an
unreachable NN cannot cause inconsistency because a fresh FBR is sent on
reconnect/re-register.
Both protections can be disabled by setting the corresponding value to
0.How was this patch tested?
New unit tests in
TestIncrementalBlockReportManager:testIBRQueueSizeLimit— queue is capped and cleared on overflow.testIBRQueueStaleInterval— queue is cleared after the stale interval.testIBRQueueNoLimit— cap disabled (0) keeps old behavior.testIBRDeduplication— per-block dedup still works.testTotalPendingIBRSizeMultipleStorages— size counting across storages.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles? N/A