HDFS-17630. Avoid PacketReceiver#MAX_PACKET_SIZE Initialized to 0.#8329
Open
balodesecurity wants to merge 2 commits intoapache:trunkfrom
Open
HDFS-17630. Avoid PacketReceiver#MAX_PACKET_SIZE Initialized to 0.#8329balodesecurity wants to merge 2 commits intoapache:trunkfrom
balodesecurity wants to merge 2 commits intoapache:trunkfrom
Conversation
The previous static initializer in PacketReceiver called new HdfsConfiguration(), which loads config resources via URL connections. If those URLs pointed to HDFS, the resulting class-loading chain could instantiate BlockReaderRemote before PacketReceiver finished initializing, causing the JVM to return the partially-initialized class with MAX_PACKET_SIZE == 0. Any subsequent block read then failed with "Incorrect value for packet payload size". Fix: initialize MAX_PACKET_SIZE directly to the compile-time default constant (removing the static initializer entirely) and introduce a static setMaxPacketSize(Configuration) method that DFSClient calls during construction to apply any configured override. This eliminates the circular class-loading dependency while preserving configurability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PacketReceiver.MAX_PACKET_SIZEwas set via astatic { new HdfsConfiguration(); ... }initializer. When config resources are loaded from HDFS URLs (e.g., during JAR loading viaURLJarFile), the resulting class-loading chain can instantiateBlockReaderRemotebeforePacketReceiverfinishes initializing. The JVM then returns the partially-initialized class withMAX_PACKET_SIZE == 0, causing every subsequent block read to throwIOException: Incorrect value for packet payload size.MAX_PACKET_SIZEdirectly to the compile-time default constant (eliminating the static initializer) and add a staticsetMaxPacketSize(Configuration)method thatDFSClientcalls during construction to apply any operator-configured override. This breaks the circular class-loading dependency while preserving configurability.Test plan
TestPacketReceiver#testMaxPacketSizeDefaultIsNonZeroandTestPacketReceiver#testSetMaxPacketSizeFromConfigpassTestPacketReceivertests continue to pass