HDFS-11039. Document 42 configuration properties missing from hdfs-default.xml. - #8637
Open
joseluisll wants to merge 1 commit into
Open
HDFS-11039. Document 42 configuration properties missing from hdfs-default.xml.#8637joseluisll wants to merge 1 commit into
joseluisll wants to merge 1 commit into
Conversation
joseluisll
force-pushed
the
HDFS-11039-hdfs-default-xml
branch
from
July 28, 2026 15:39
c82f036 to
fca950f
Compare
…fault.xml.
Adds documentation for 42 live, non-deprecated configuration properties
that are read by production code but were absent from hdfs-default.xml:
16 NFS gateway properties, 16 encrypted-transfer allow/deny-list
properties, 3 WebHDFS OAuth2 properties, 1 standby edit-tailing
property, 1 short-circuit read metrics property, and 5 properties that
were excluded from TestHdfsConfigFields under stale skip-list reasons
("Fully deprecated properties?", "Removed by HDFS-6440") that code
inspection disproves.
The only Java change is skip-list bookkeeping in TestHdfsConfigFields,
following the file's existing idiom (cf. the "nfs" prefix entry):
stale skip entries removed so the reflected constants are enforced
against the xml again, and skip entries added for properties whose
constants live outside the classes the test reflects over.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
joseluisll
force-pushed
the
HDFS-11039-hdfs-default-xml
branch
from
July 28, 2026 15:41
fca950f to
7337373
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Documents 42 configuration properties in
hdfs-default.xmlthat are readby production code but were previously discoverable only by reading the
source. Every property listed below is live and non-deprecated: each read
site was verified, none appears in any
DeprecationDeltatable, and nonecarries a deprecated constant. This answers the question asked on HDFS-11039
in 2018 — which properties should be added.
No production code is changed. The diff touches exactly two files:
hdfs-default.xml(+42<property>entries, 1 rename of a deprecatedproperty name to its current one) and
TestHdfsConfigFields.java(skip-listbookkeeping only, explained below).
The 42 properties
dfs.datanode.synconclosefalseBlockReceiver. Was skip-listed under// Fully deprecated properties?— disproved.dfs.datanode.non.local.lazy.persistfalseDNConf. Was skip-listed under the same stale comment — disproved.dfs.namenode.tolerate.heartbeat.multiplier4BlockPlacementPolicyDefault. Same stale skip entry — disproved.dfs.namenode.replqueue.threshold-pctdfs.namenode.safemode.threshold-pct)BlockManagerSafeMode. Same stale skip entry — disproved.dfs.ha.log-roll.rpc.timeout20000EditLogTailer. Was skip-listed as// Removed by HDFS-6440— the property is still read; the claim is incorrect.dfs.ha.tail-edits.max-txns-per-lock9223372036854775807EditLogTailer, outside any*ConfigKeysclass.dfs.client.read.shortcircuit.metrics.sampling.percentage0HdfsClientConfigKeys.Read.ShortCircuit, which the comparison test does not reflect over.dfs.webhdfs.oauth2.credentialCredentialBasedAccessTokenProvider. Siblings were documented; this constant lives outsideHdfsClientConfigKeys.dfs.webhdfs.oauth2.refresh.tokenConfRefreshTokenBasedAccessTokenProvider. Same situation.dfs.webhdfs.oauth2.refresh.token.expires.ms.since.epochnfs.http.port50079nfs.http.address.nfs.http.address0.0.0.0:50079nfs.https.port50579nfs.https.address.nfs.https.address0.0.0.0:50579nfs.file.dumptruenfs.dtmax65536nfs.export.point/nfs.max.open.files256nfs.stream.timeout600000nfs.large.file.uploadtruenfs.aix.compatibility.mode.enabledfalsenfs.metrics.percentiles.intervalsnfs.superusernfs.registration.port40nfs.udp.client.portmap.timeout.millis500nfs.port.monitoring.disabledtruenfs.allow.insecure.portsperNfsConfiguration'sDeprecationDelta.dfs.datatransfer.server.fixedBlackList.file/etc/hadoop/fixedBlackListBlackListBasedTrustedChannelResolver.dfs.datatransfer.server.variableBlackList.enablefalsedfs.datatransfer.server.variableBlackList.file/etc/hadoop/blackListdfs.datatransfer.server.variableBlackList.cache.secs3600dfs.datatransfer.client.fixedBlackList.filedfs.datatransfer.client.variableBlackList.enablefalsedfs.datatransfer.client.variableBlackList.filedfs.datatransfer.client.variableBlackList.cache.secs3600dfs.datatransfer.server.fixedwhitelist.file/etc/hadoop/fixedwhitelistWhitelistBasedTrustedChannelResolver.dfs.datatransfer.server.variablewhitelist.enablefalsedfs.datatransfer.server.variablewhitelist.file/etc/hadoop/whitelistdfs.datatransfer.server.variablewhitelist.cache.secs3600dfs.datatransfer.client.fixedwhitelist.filedfs.datatransfer.client.variablewhitelist.enablefalsedfs.datatransfer.client.variablewhitelist.filedfs.datatransfer.client.variablewhitelist.cache.secs3600Why TestHdfsConfigFields is in the diff
TestHdfsConfigFieldsenforces, in both directions, thathdfs-default.xmland the reflected
*ConfigKeysclasses agree, modulo its skip lists.Documenting these properties therefore requires skip-list bookkeeping — the
same idiom as the existing
nfsprefix entry, routinely updated by pastpatches (HDFS-15870, HDFS-15514, HDFS-15559):
dfs.datanode.synconclose,dfs.datanode.non.local.lazy.persist,dfs.namenode.tolerate.heartbeat.multiplier,dfs.namenode.replqueue.threshold-pct,dfs.ha.log-roll.rpc.timeout), sothose constants are again enforced against the xml and the gap cannot
silently reopen for them.
dfs.datatransfer.{server,client}.*,dfs.ha.tail-edits.max-txns-per-lock,dfs.webhdfs.oauth2.*anddfs.client.read.shortcircuit.metrics.sampling.percentage, whose constantslive in classes the test does not reflect over. The
nfs.*entries arecovered by the existing
nfsprefix entry;NfsConfigKeysitself cannot bereflected, as
hadoop-hdfs-nfsdepends onhadoop-hdfsand can never be onthis test's classpath.
skip-listed (
dfs.corruptfilesreturned.max,dfs.metrics.session-id).A follow-up can strengthen enforcement by promoting the declaring classes
into
configurationClasseswhere module dependencies allow.Method — how the list was produced and verified
The list was produced by static analysis of the entire
hadoop-hdfs-projectsource tree, cross-validated by two independently implemented scanners (one
Java, one Python), and verified against Hadoop's own machinery:
Stringconfig-key constant in every class(not just
*ConfigKeysregistries), resolving multi-line declarations andPREFIX + "suffix"concatenation, including cross-class references.Configurationaccessor call site(
get*,getTimeDuration,getPropsWithPrefix, …) resolved back to akey, catching literal-only keys that have no constant at all.
DeprecationDeltatables(
HdfsConfiguration,NfsConfiguration),HdfsClientConfigKeys.DeprecatedKeys,and
@Deprecatedannotations. A@Deprecatedconstant was not treatedas a deprecated property (most such constants merely moved classes).
Test*ConfigFieldsskip entry re-checkedagainst the code. Reasons stating intent ("not intended for users",
"purposely hidden") were respected; reasons stating checkable facts
("deprecated", "removed") were verified.
required to reproduce the green
TestHdfsConfigFields/TestRBFConfigFieldsresults with a zero diff; any discrepancy was treated as a scanner bug.
HdfsConfigurationwas instantiated and its mergedproperty set diffed against the xml through the real
Configuration.loadResourcepipeline, including${...}resolution.Descriptions policy. Descriptions are derived from the declaring
constant's Javadoc, surrounding code comments, or direct inspection of the
read site — never invented. Secrets and unset-by-default properties use an
empty
<value>, matching file convention.Properties deliberately not documented — exclusions and their justification
The audit identified further undocumented keys that are deliberately excluded
from this patch. Each exclusion is justified by explicit source-level intent,
by deprecation status, or by demonstrated absence of effect:
dfs.namenode.snapshot.deletion.ordered,dfs.namenode.snapshot.deletion.ordered.gc.period.ms,dfs.namenode.snapshot.trashroot.enabledSnapshotManager,FSNamesystem) designate these keys as private configuration. Documenting them would contradict recorded upstream intent.dfs.ha.tail-edits.qjm.rpc.max-txnsQuorumJournalManageras not publicly exposed.dfs.datanode.xceiver.stop.timeout.millisDFSConfigKeysas hidden and intentionally undocumented.dfs.datanode.startup,dfs.namenode.startupignore.secure.ports.for.testing,dfs.datanode.enable.fileio.fault.injection,dfs.datanode.duplicate.replica.deletiondfs.datanode.min.supported.namenode.version,dfs.namenode.min.supported.datanode.versiondfs.corruptfilesreturned.maxdfs.metrics.session-idHdfsClientConfigKeys.DeprecatedKeys. The skip-list comment has been corrected to record this.dfs.namenode.audit.log.asyncdfs.federation.router.*(4 keys)hdfs-rbf-default.xml; a separate JIRA will be filed.Contains content generated by Anthropic Claude Code.
How was this patch tested?
mvn test -Dtest=TestHdfsConfigFieldson the rebased branch againstcurrent trunk (JDK 17): 4/4 pass, BUILD SUCCESS.
duplicate
dfs.journalnode.kerberos.*entries were noticed upstream;untouched here as out of scope.)
HdfsConfigurationand diffed its mergedproperty set against the xml through the real resource-loading pipeline;
every non-empty documented value matches the code default, including
${...}resolution.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles? (N/A)AI Tooling
This contribution was prepared with the assistance of Anthropic Claude Code:
property discovery, classification and description drafting were performed
with its assistance and were reviewed, verified against the source, and
approved by the contributor.
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html