September 2023 APAC Community Sync

[ChenSammi]

Sep 1st, 2023

Attender: Guohao, Minyu, Yiyang, XiChen, Hongbing, Guangbao, Hualong, Yuanben, Sammi
Agenda:

1. Guohao shared the experience about how to migrate Container Schema V2 data to Schema V3, the problem they met and how it’s resolved. In their last try,, it took around 15 min to migrate a disk volume(8TB, 60% utilization) data from V2 to V3.

2. Hongbin and Guangbao noticed that after a very high data ingestion workload, the Datanode instance will occupy almost all the memory on this server, mostly were off-heap memory usages. And after the workload stopped, the memory consumption of the Datanode instance didn’t shrink obviously. There is potentially an off-heap memory leak here. They will continue to monitor the service and collect more data/info for further investigation.

3. 1.4.0 Release

   • RATIS 3.0.0 enabling in Ozone is ongoing. There are some snapshot related CI failures and need fixing.
   • Yiyang has filtered all open blockers and critical issues which targets 1.4.0. Most of them are snapshot improvement and certificate rotation patches.

4. For “Missing” containers shown on recon, Guangbao would like to know how to identify if there is real data missing or not. One way is to search this container ID in OM DB, to see if there is any file/object whose block location contains the container ID. For the coming 1.4.0 release, there are new features on Recon which help to further analyze these “Missing” container cases.

[ChenSammi]

Sep 15th, 2023
Attender: Guo Hao, Bing Hong, George Huang, Xi Chen, Guangbao Zhao, Jackson Yao, Hualong Zhang, Minyu Liu, Yuanben Wang, Sammi Chen

Agenda:

1. Goerge raised the question about 1.3.1 release. After the discuss, we reach the consensus that it's better to put all our resource on 1.4.0 to speed it up. Once 1.4.0 is released, most of partners have the plan to upgrade to it. George will send a notice in the community dev mail list later about that we are going to cancel the 1.3.1 release.
2. Bing raised a question about block deletion. They have deleted a lot of files recently， and find that the block deletion is still going on after several days. Turing the block deletion related properties of OM, SCM and DN, doesn't help. Finally it's found that because there are too closeContainerCommand in the queue and the thread is busy handling these commands, so the block deletion commands are waiting for a long time to get executed. Please refer to the https://docs.google.com/document/d/1g1h-63fvA-be-clvyVRAHLWehoadCnjNRWyX8Bp-UIU/edit#heading=h.358h6n518zuj for detail info.
3. Xi shared their block deletion turning experience，after some turning，1K blocks deletion per second can be achieved on one DN, and improvements proposed to community based on the real environment cases
   HDDS-8888. Consider Datanode queue capacity when sending DeleteBlocks command #4939
   HDDS-8869. Make DN DeleteBlocksCommandHandler wait for the lock can timeout. #4913
   HDDS-8690. Ozone Support deletion related parameter dynamic configuration #4798
   HDDS-8882. Manage status of DeleteBlocksCommand in SCM to avoid sending duplicates to Datanode #4988
   Review help is needed for above patches
4. Xi also mentioned that when the SCM rocksdb has 2 billion keys in the delete transaction table，it took nearly 60-70s for the rocksdb iterator to seek to first place.
5. Hao found their SCM failed to startup because of raft log gap. And it's very hard for SCM to recover from that state without any SCM metadata loss. He raised ticket RATIS-1887.

[errose28]

Finally it's found that because there are too closeContainerCommand in the queue and the thread is busy handling these commands, so the block deletion commands are waiting for a long time to get executed.

This will likely be fixed by #5277

[ChenSammi]

Sep 22th, 2023
Attender: Ivan, Siddhant, Sammi

Agenda:

1. DiDi reported that they have more than 74K replicated container commands on one DN. After merged HDDS-9122, the problem is gone.
2. Ivan raised some questions about the ReplicationManager and LegacyReplicatioManager. Siddhant explained that the LegacyReplicatioManager is replaced by ReplicatioManager now in the upstream. Siddhant will also share the new ReplicatioManager related documents to community later.