HDDS-16171. Tool to download OM metadata with snapshots. - #11017
HDDS-16171. Tool to download OM metadata with snapshots.#11017sadanand48 wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a new ozone repair om download CLI subcommand (in cli-repair) that downloads and reconstructs an OM RocksDB checkpoint by reusing the follower-bootstrap snapshot transfer flow (OmRatisSnapshotProvider), along with integration coverage validating end-to-end behavior against a MiniOzone HA cluster.
Changes:
- Added
DownloadOMDBrepair tool and wired it into the OM repair CLI subcommands. - Implemented snapshot download + local construction of an
om.dboutput directory via the follower-bootstrap checkpoint transfer path (forcing inode-based/v2 transfer). - Added integration tests (
TestDownloadOMDBTool) covering successful download and overwrite behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/repair/om/TestDownloadOMDBTool.java | Adds HA MiniOzone integration coverage for the new ozone repair om download command, including overwrite semantics and v2 endpoint usage validation. |
| hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/OMRepair.java | Registers the new download subcommand under OMRepair. |
| hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java | Implements the new repair CLI tool to download and construct OM DB from a specified OM node using follower-bootstrap checkpoint transfer. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Move the full follower-bootstrap checkpoint directory to --output-dir so both om.db and db.snapshots are retained. Add integration coverage that creates an Ozone snapshot and verifies its checkpoint files are present in the downloaded metadata layout. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I06f561777bfd33cd63b6ae336f4e1c28bea02653
Wrap OmRatisSnapshotProvider log capture in try/finally and call stopCapturing() to avoid leaking Log4j appenders into other tests. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I86270febe5b1682da33bf5fe24ed63d465f88335
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
Suppressed comments (4)
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:89
- In an HA configuration, omitting the optional
--service-idleavesomServiceIdnull, so this lookup searchesozone.om.address.<node-id>instead of the configuredozone.om.address.<service-id>.<node-id>and the command fails even when exactly one service is configured. Resolve the default service ID before looking up the node; the existing helper also gives a clear error when multiple service IDs make the omission ambiguous.
OMNodeDetails omNodeDetails =
OMNodeDetails.getOMNodeDetailsFromConf(conf, omServiceId, nodeId);
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:96
DownloadOMDBinherits the global--dry-runoption, but this method never checks it. As a result,--dry-run --overwritestill deletes the existing output and performs the download, contradicting the option's promise not to make changes. Return before the filesystem mutations and network transfer when dry-run is enabled.
if (Files.exists(outputDir)) {
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:100
- This validation failure prints to stderr and then returns normally from the
Callable, so Picocli reports exit code 0; the new overwrite test currently confirms that behavior. Automation will treat a refused download as successful. Fail with a non-zero status (for example viafatal) and apply the same correction to the earlier HA/node-resolution validation returns, then assert a non-zero code in the test.
if (!overwrite) {
error("Output directory already exists: %s. Use --overwrite to replace it.",
outputDir.toAbsolutePath());
return;
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:112
- The work directory is created on the JVM's temporary filesystem, while the potentially very large final metadata may be on another filesystem.
OmRatisSnapshotProvidertherefore checks free space on the wrong volume, andmoveDirectorymay fall back to copying every file across filesystems, losing hard-link space savings and leaving a partial output if the destination fills. Create the work directory beside the output so the provider checks the destination volume and the final move remains on one filesystem.
Path snapshotWorkDir = Files.createTempDirectory("ozone-omdb-bootstrap-");
| try (OmRatisSnapshotProvider provider = new OmRatisSnapshotProvider( | ||
| conf, snapshotWorkDir.toFile(), | ||
| Collections.singletonMap(omNodeDetails.getNodeId(), omNodeDetails))) { | ||
| checkpoint = provider.downloadDBSnapshotFromLeader(omNodeDetails.getNodeId()); |
| outputDir.toAbsolutePath()); | ||
| return; | ||
| } | ||
| FileUtils.forceDelete(outputDir.toFile()); |
What changes were proposed in this pull request?
This PR adds a new OM repair CLI tool to download and construct om.db using the same follower bootstrap transfer path.
Added ozone repair om download (DownloadOMDB) in ozone-cli-repair.
The command reuses OmRatisSnapshotProvider / follower-bootstrap checkpoint flow and forces v2 inode-based transfer (ozone.om.db.checkpoint.use.inode.based.transfer=true) for this operation.
Added CLI registration under OMRepair.
Replaced the earlier mocked unit coverage with an integration test (TestDownloadOMDBTool) that validates end-to-end behavior against a MiniOzone HA cluster:
successful download/build of om.db
v2 checkpoint endpoint usage
overwrite behavior for existing output directory
Why
Operators need a practical offline/repair utility to fetch and assemble a usable om.db exactly the way OM followers bootstrap, instead of maintaining separate transfer logic.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16171
How was this patch tested?
Added/updated integration coverage:
TestDownloadOMDBTool#testDownloadConstructsOmDbUsingV2Transfer
TestDownloadOMDBTool#testOverwriteReplacesExistingOutput