HDDS-16120. A single MiniOzoneCluster build timeout cascades into whole-suite failures in MiniOzoneClusterProvider

[echonesis]

What changes were proposed in this pull request?

MiniOzoneClusterProvider creates clusters on a background thread and passes them to consumers through a bounded blocking queue. Previously, an IOException or TimeoutException during cluster creation terminated the background thread. The queue was then never refilled, causing every subsequent provide() call to wait for 100 seconds and fail with the misleading Failed to obtain available cluster in time message.

This change passes both successful clusters and build failures through the existing bounded queue:

• A failed build is surfaced immediately from the affected provide() call, with the original exception preserved as its cause.
• The background create thread continues building clusters after a failure.
• A partially created cluster is shut down before the failure is published.
• Subsequent provide() calls can receive successfully created clusters.
• Failed builds do not count toward the configured cluster limit.

The queue capacity remains one and put() is blocking, providing natural backpressure when builds fail persistently. This prevents the create thread from processing failures in an unbounded tight loop without introducing a retry limit, backoff duration, or polling interval.

The change limits the blast radius of an individual cluster startup failure. It does not attempt to address the underlying cause of slow cluster startup.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16120

How was this patch tested?

Local Test

mvn -pl :ozone-mini-cluster test \
    -Dtest=TestMiniOzoneClusterProvider \
    -DskipShade -DskipRecon -DskipDocs

GitHub Actions CI: https://github.com/echonesis/ozone/actions/runs/31478614848

Generated-by: Codex (GPT-5)

[chihsuan]

Thanks for the patch! @echonesis Passing failures through the existing bounded queue is a nice approach.

I left a few small comments around cleanup.

[chihsuan]

If an interrupt was already consumed during cluster creation or cleanup, put() can block on a full queue while shutdown() waits in join(). Would a bounded offer() be safer?

[chihsuan]

Just a thought: This cleanup may be interrupted before it completes. It runs on the create thread, which provider shutdown interrupts, while the reaper thread intentionally avoids interrupting cluster cleanup

The branch above does the same thing, so it may be fine to leave it as is. Would routing through expiredClusters be worth considering?