Skip to content

HDDS-16170. Intermittent failure in TestSnapshotDiffManager#testLoadJobsOnStartUp - #11014

Draft
smengcl wants to merge 1 commit into
apache:masterfrom
smengcl:HDDS-16170
Draft

HDDS-16170. Intermittent failure in TestSnapshotDiffManager#testLoadJobsOnStartUp#11014
smengcl wants to merge 1 commit into
apache:masterfrom
smengcl:HDDS-16170

Conversation

@smengcl

@smengcl smengcl commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 4.8)

What changes were proposed in this pull request?

TestSnapshotDiffManager#testLoadJobsOnStartUp fails intermittently with expected: <DONE> but was: <IN_PROGRESS>.

The test stubs generateSnapshotDiffReport with a doAnswer that writes DONE to the DB inside the answer body, calls loadJobsOnStartUp(), then waits with attempt(() -> verify(spy, atLeast(1)).generateSnapshotDiffReport(...)) before reading the job back and asserting DONE.

Mockito records an invocation before it runs the stubbed answer, so the verify(...) wait can pass as soon as the worker thread enters generateSnapshotDiffReport, while the answer is still running and has not yet persisted DONE. The main thread then reads the job and observes the original IN_PROGRESS, so the assertion fails.

The fix waits on the actual observable end state the test asserts: it polls getSnapshotDiffJobFromDb(...).getStatus() until it is DONE, instead of waiting on the Mockito invocation. This is deterministic by construction because the read the assertion depends on is exactly what is waited on. No production code changes.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16170

How was this patch tested?

Ran the affected test locally against the change:

mvn -pl :ozone-manager -am test -Dtest='TestSnapshotDiffManager#testLoadJobsOnStartUp' -DskipShade -DskipRecon -DskipDocs -Dsurefire.failIfNoSpecifiedTests=false

A single green run cannot prove non-flakiness, so the fix is made deterministic by construction: the wait polls the same persisted job status that the subsequent assertion reads, closing the window between Mockito recording the invocation and the stubbed answer writing DONE to the DB.

…obsOnStartUp

testLoadJobsOnStartUp stubs generateSnapshotDiffReport with a doAnswer that
writes DONE to the DB inside the answer body, then waits on
verify(spy, atLeast(1)).generateSnapshotDiffReport(...) before reading the job
back and asserting DONE.

Mockito records an invocation before it runs the stubbed answer, so the verify
wait can pass while the answer is still executing and has not yet persisted
DONE. The subsequent DB read then observes the original IN_PROGRESS and the
assertion fails intermittently (expected: <DONE> but was: <IN_PROGRESS>).

Wait on the observable end state the test asserts instead: poll the persisted
job status until it is DONE. This closes the window between Mockito recording
the invocation and the answer writing DONE, making the test deterministic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 13, 2026 07:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an intermittent failure in TestSnapshotDiffManager#testLoadJobsOnStartUp by changing the test’s synchronization strategy to wait on the persisted job status rather than a Mockito invocation, eliminating a race between method entry and the stubbed doAnswer completing its DB write.

Changes:

  • Replace attempt(verify(...)) with polling of getSnapshotDiffJobFromDb(...).getStatus() until it reaches DONE.
  • Add explanatory comments documenting why Mockito-based waiting was flaky in this scenario.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jojochuang jojochuang added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-gen snapshot https://issues.apache.org/jira/browse/HDDS-6517 test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants