Skip to content

HDDS-16171. Tool to download OM metadata with snapshots. - #11017

Draft
sadanand48 wants to merge 4 commits into
apache:masterfrom
sadanand48:HDDS-16171
Draft

HDDS-16171. Tool to download OM metadata with snapshots.#11017
sadanand48 wants to merge 4 commits into
apache:masterfrom
sadanand48:HDDS-16171

Conversation

@sadanand48

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds a new OM repair CLI tool to download and construct om.db using the same follower bootstrap transfer path.

Added ozone repair om download (DownloadOMDB) in ozone-cli-repair.
The command reuses OmRatisSnapshotProvider / follower-bootstrap checkpoint flow and forces v2 inode-based transfer (ozone.om.db.checkpoint.use.inode.based.transfer=true) for this operation.
Added CLI registration under OMRepair.
Replaced the earlier mocked unit coverage with an integration test (TestDownloadOMDBTool) that validates end-to-end behavior against a MiniOzone HA cluster:
successful download/build of om.db
v2 checkpoint endpoint usage
overwrite behavior for existing output directory

Why
Operators need a practical offline/repair utility to fetch and assemble a usable om.db exactly the way OM followers bootstrap, instead of maintaining separate transfer logic.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16171

How was this patch tested?

Added/updated integration coverage:
TestDownloadOMDBTool#testDownloadConstructsOmDbUsingV2Transfer
TestDownloadOMDBTool#testOverwriteReplacesExistingOutput

@sadanand48 sadanand48 added snapshot https://issues.apache.org/jira/browse/HDDS-6517 AI-gen labels Aug 13, 2026
@jojochuang
jojochuang requested review from jojochuang and a lite review from Copilot August 13, 2026 15:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new ozone repair om download CLI subcommand (in cli-repair) that downloads and reconstructs an OM RocksDB checkpoint by reusing the follower-bootstrap snapshot transfer flow (OmRatisSnapshotProvider), along with integration coverage validating end-to-end behavior against a MiniOzone HA cluster.

Changes:

  • Added DownloadOMDB repair tool and wired it into the OM repair CLI subcommands.
  • Implemented snapshot download + local construction of an om.db output directory via the follower-bootstrap checkpoint transfer path (forcing inode-based/v2 transfer).
  • Added integration tests (TestDownloadOMDBTool) covering successful download and overwrite behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/repair/om/TestDownloadOMDBTool.java Adds HA MiniOzone integration coverage for the new ozone repair om download command, including overwrite semantics and v2 endpoint usage validation.
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/OMRepair.java Registers the new download subcommand under OMRepair.
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java Implements the new repair CLI tool to download and construct OM DB from a specified OM node using follower-bootstrap checkpoint transfer.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

jojochuang and others added 3 commits August 13, 2026 18:03
Move the full follower-bootstrap checkpoint directory to --output-dir
so both om.db and db.snapshots are retained. Add integration coverage
that creates an Ozone snapshot and verifies its checkpoint files are
present in the downloaded metadata layout.

Co-authored-by: Cursor <cursoragent@cursor.com>
Change-Id: I06f561777bfd33cd63b6ae336f4e1c28bea02653
Wrap OmRatisSnapshotProvider log capture in try/finally and call
stopCapturing() to avoid leaking Log4j appenders into other tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
Change-Id: I86270febe5b1682da33bf5fe24ed63d465f88335

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:89

  • In an HA configuration, omitting the optional --service-id leaves omServiceId null, so this lookup searches ozone.om.address.<node-id> instead of the configured ozone.om.address.<service-id>.<node-id> and the command fails even when exactly one service is configured. Resolve the default service ID before looking up the node; the existing helper also gives a clear error when multiple service IDs make the omission ambiguous.
    OMNodeDetails omNodeDetails =
        OMNodeDetails.getOMNodeDetailsFromConf(conf, omServiceId, nodeId);

hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:96

  • DownloadOMDB inherits the global --dry-run option, but this method never checks it. As a result, --dry-run --overwrite still deletes the existing output and performs the download, contradicting the option's promise not to make changes. Return before the filesystem mutations and network transfer when dry-run is enabled.
    if (Files.exists(outputDir)) {

hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:100

  • This validation failure prints to stderr and then returns normally from the Callable, so Picocli reports exit code 0; the new overwrite test currently confirms that behavior. Automation will treat a refused download as successful. Fail with a non-zero status (for example via fatal) and apply the same correction to the earlier HA/node-resolution validation returns, then assert a non-zero code in the test.
      if (!overwrite) {
        error("Output directory already exists: %s. Use --overwrite to replace it.",
            outputDir.toAbsolutePath());
        return;

hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java:112

  • The work directory is created on the JVM's temporary filesystem, while the potentially very large final metadata may be on another filesystem. OmRatisSnapshotProvider therefore checks free space on the wrong volume, and moveDirectory may fall back to copying every file across filesystems, losing hard-link space savings and leaving a partial output if the destination fills. Create the work directory beside the output so the provider checks the destination volume and the final move remains on one filesystem.
    Path snapshotWorkDir = Files.createTempDirectory("ozone-omdb-bootstrap-");

Comment on lines +114 to +117
try (OmRatisSnapshotProvider provider = new OmRatisSnapshotProvider(
conf, snapshotWorkDir.toFile(),
Collections.singletonMap(omNodeDetails.getNodeId(), omNodeDetails))) {
checkpoint = provider.downloadDBSnapshotFromLeader(omNodeDetails.getNodeId());
outputDir.toAbsolutePath());
return;
}
FileUtils.forceDelete(outputDir.toFile());
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-gen snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants