HDDS-16044. SCM should forward Datanode's current version to clients for read and write operations

[errose28]

DRAFT Builds on #11006

What changes were proposed in this pull request?

HDDS-16154 has Datanode's select their current version to send to clients based on whether or not they are finalized for ZDU and can use the unified versioning framework. SCM must pass this version through in the Pipeline + DatanodeDetails objects given to clients for read and write operations to datanodes. Prior to this, the current version assigned by the datanodes represented their software version, which was static as long as the process was running with the same software.

The complication with this change is that SCM currently stores Datanode metadata in three places:

• Persisted in PipelineManager
  • Only assigned on pipeline creation and never updated afterwards
• In-memory in NodeManager
  • Always has the most up to date information from the last heartbeat
• In-memory as replicas in ContainerManager
  • Happens to have the most up to date information because replicas hold references to the same objects in NodeManager

For write operations and reads from open containers, SCM was sourcing currentVersion from the persisted PipelineManager metadata. Prior to ZDU, this would return outdated version information for pipelines that remained after an upgrade when Datanode version increased. With ZDU this will also be a problem since the currentVersion will be updated on finalization.

For read operations from closed containers, it was sourcing currentVersion from the container replicas in ContainerManager. The ContainerManager replicas happened to have references to the NodeManager DatanodeInfo objects so they were returning up to date information, although there is nothing enforcing that this will remain true going forward.

To account for these issues, this PR substitutes the currentVersion from NodeManager at the time of serialization in the translator layers before returning the information back to the client. This avoids any invasive changes to the storage schema and any extra copies to update immutable objects like Pipeline. The downside is that it must be inserted into every operation that returns DatanodeDetails to the client, although the surface area is small. Currently this is:

• ScmBlockLocationProtocolServerSideTranslatorPB#allocateScmBlock
  • Write request, requires the minimum common version among all involved datanodes.
  • Handling from HDDS-15641. SCM should send write pipeline version with DatanodeDetails on block allocation #10878 was refactored but overall approach is the same.
• StorageContainerLocationProtocolServerSideTranslatorPB#allocateContainer, #getContainerWithPipeline, #getContainerWithPipelineBatch, #getExistContainerWithPipelinesInBatch
  • Read requests, all datanode currentVersions can be forwarded exactly.
  • All read requests return a ContainerWithPipeline object.
    • ContainerWithPipeline#getProtobuf has been updated to require an explicit version be provided for each replica by the caller, ensuring all read requests are covered.

What is the link to the Apache JIRA

HDDS-16044

How was this patch tested?

• Unit test added for each API in the translator layers responsible for forwarding the version information.
• Integration test added for end to end testing that all current versions sent by the datanode end up at the client for the matrix of read/write+EC/Ratis+open/closed containers.