HDDS-16190. Do not use unified version framework in OM until ZDU is finalized.

[errose28]

DRAFT Builds on #11006

What changes were proposed in this pull request?

Background

ZDU is the first version that supports the unified versioning framework, where network and disk compatibility are tracked with the same version. Therefore, we cannot use this framework before ZDU is finalized. The existing VersionManager already handles this for disk compatibility. It will read from the old layout feature enums until ZDU is finalized. We need similar handling for the component version sent over the network to clients. We cannot send the unified version framework's apparent version to clients until ZDU is finalized, otherwise we may be sending them a layout feature when they are expecting a component version. When ZDU is pre-finalized, we must send the last component version before ZDU to clients. Finalizing ZDU moves both version systems ahead to the unified one that starts with ZDU, where one apparent version can be shared over both disk and network.

This has been handled for Datanodes in #11006, which also added the common getVersionForClient method and unit test we will use for OM as well.

What is the link to the Apache JIRA

HDDS-16190

How was this patch tested?

• Existing unit test in HDDS-16154. Datanode should assign DatanodeDetails#currentVersion based on its apparent version #11006 covers the version returned by OmVersionManager
• Integration test added to check the version returned in a live HA cluster before and after finalization.

[ss77892]

A couple comments from my second brain:
(1) with the setCurrentVersion call removed from HddsDatanodeService.initializeDatanodeDetails(), it looks like nothing sets currentVersion before the datanode registers. On a fresh start the builder now defaults to DEFAULT_VERSION, on a restart the value in datanode.id is dropped because the yaml setter is now a no-op, and the only remaining write happens in HeartbeatEndpointTask, which runs after registration. Since RegisterEndpointTask serializes the details object directly, every registration ends up advertising DEFAULT_VERSION to SCM until the first heartbeat corrects it, a short window, but it briefly reports an even older version than the pre-ZDU.
(2) A small follow-up on the HeartbeatEndpointTask change: the task now mutates the shared DatanodeDetails via setCurrentVersion on every call(), and since the endpoint executor runs one thread per SCM plus Recon, several tasks can do this concurrently. currentVersion used to be write-once at startup, but with ZDU it's now intentionally updated at runtime, which puts it in the same category as persistedOpState and persistedOpStateExpiryEpochSec, both volatile for exactly that reason. Without volatile there's no visibility guarantee between threads, so one endpoint could keep reporting a stale version after another already updated it. Could we mark currentVersion volatile to match?

[errose28]

Thanks for the review. These look valid for #11006 so I'll fix them there. There's a large chain of stacked PRs targeting the ZDU branch right now. Following the draft links backwards gives this layout:

#11025 -> #11006 -> #10994
          ^
         /
#11023 -