HDDS-16116. Merge recent commits from master to HDDS-11233

[devmadhuu]

What changes were proposed in this pull request?

Merge recent commits from master into the HDDS-11233 feature branch to keep it in sync with mainline. This brings the branch up to date with master and reconciles the StorageTier feature commits with recent master changes.

The feature branch's original commit history is preserved (first-parent), with a single Merge remote-tracking branch 'upstream/master' into HDDS-11233 commit on top.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16116

How was this patch tested?

Verified core modules compile against the merged master locally; relying on this PR's CI to validate before merge. To be merged via GitHub after CI passes and review.

Made with Cursor

[devmadhuu]

@ivandika3 @jojochuang @ChenSammi @xichen01 @ashishkumar50 Kindly review this merge PR for merging master to feature branch for storage policy.

[ivandika3]

@devmadhuu Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to HDDS-11233.

[devmadhuu]

Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to HDDS-11233.

Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:

1. SCMCommonPlacementPolicy#hasEnoughSpace — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.
2. Pipeline providers (RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.
3. BlockManagerImpl#allocateBlock — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.
4. ContainerProtocolCalls#writeChunkAsync / BlockOutputStream — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.
5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)
6. NodeUtils — updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.
7. Proto (OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto) — feature's storage-tier/policy fields (e.g. OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy) . All additive, no renumbering of existing fields.
8. EC/Ratis under-replication handlers (ECUnderReplicationHandler, RatisUnderReplicationHandler)

[devmadhuu]

Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to HDDS-11233.

Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:

1. SCMCommonPlacementPolicy#hasEnoughSpace — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.
2. Pipeline providers (RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.
3. BlockManagerImpl#allocateBlock — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.
4. ContainerProtocolCalls#writeChunkAsync / BlockOutputStream — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.
5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)
6. NodeUtils — updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.
7. Proto (OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto) — feature's storage-tier/policy fields (e.g. OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy) . All additive, no renumbering of existing fields.
8. EC/Ratis under-replication handlers (ECUnderReplicationHandler, RatisUnderReplicationHandler)

Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to HDDS-11233.

Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:

1. `SCMCommonPlacementPolicy#hasEnoughSpace` — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.

2. Pipeline providers (`RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider`) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.

3. `BlockManagerImpl#allocateBlock` — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.

4. `ContainerProtocolCalls#writeChunkAsync / BlockOutputStream` — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.

5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)

6. `NodeUtils `— updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.

7. Proto (`OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto`) — feature's storage-tier/policy fields (e.g. `OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy`) . All additive, no renumbering of existing fields.

8. EC/Ratis under-replication handlers (`ECUnderReplicationHandler, RatisUnderReplicationHandler`)

@ivandika3 did you get chance ? I am working on green CI.