2024 Jan APAC Community Sync

[ChenSammi]

Jan 5th
Attenders: Guohao, Yiyang, Jianghua, Hualong, Yuanben, Hongbin, Kangchen, Ce, ..., Sammi

1. Shopee

   1. 1.4.0 RC0 VOTE is ongoing. Welcome the community contributors to evaluate the RC0 package and vote in the thread.

2. DiDi

   1. Doing OM HA performance evaluate. Plan to migrate OM from non HA to HA if performance meets the requirement.
   2. Face the same objectID issue after previously convert from OM HA to non HA. The current objectID and updateID are strictly related with ratis transaction ID. If for any reason, the ratis need to be reset and transaction ID will start from 0 again, then it will cause updateID compare issue and duplicate object ID issue after that. One case could be, use a backed OM rocksdb directory to start new OM instances for a new OM namespace. A ratis independent incrementally increased ID is one solution.
   3. Suggest to provide a ozone command to manual update tx ID in OM rocksdb in case the tx ID is not updated due to any potential bugs in OM.
   4. Need help review on
      HDDS-9988. SCM UI shows storage usage percentage #5882

3. Qihoo

   1. Find one bug in GrpcOmTransport, that GrpcOmTransport cannot welly handle the exception once OM leader switches.
   2. Working on a OM block batch allocation proposal to reduce the request from OM to SCM. With the approach, the SCM block allocation request can reduce by 90%.
   3. Benchmarked SCM block allocation QPS with a 36 Core server, it's about 20K QPS. Would like to know if there is any way to improve the QPS from 20K.

4. China Unicom

   1. Start to use ozone.
   2. Find one issue when using 1.3.0 hadoop3 ozone client with Spark 3.
   3. Hadoop 2.7 currently works with Ozone. But Hadoop 2.7 support is dropped sometime before. So the compatibility cannot be full guaranteed in later Ozone version.
   4. Can use distcp to migrate data from hadoop 2.7 to Ozone cluster.

[arp7]

Thanks for writing up the report @ChenSammi.

The current objectID and updateID are strictly related with ratis transaction ID. If for any reason, the ratis need to be reset and transaction ID will start from 0 again, then it will cause updateID compare issue and duplicate object ID issue after that.

Is this a serious issue during non-HA to HA migration? Can it also occur without this migration?

[ChenSammi]

It's a serious issue during non-HA to HA, or HA to non-HA change. It will not happen if there is no such migration, except one case that if user uses the rocksdb data of an existing Ozone cluster as a base, to start a ozone cluster on top of that.

[whbing]

The related patch : #6281

[ChenSammi]

Jan 19th
Attenders: Guohao, Minyu, Yiyang, Xi, Hongbin, Jianghua, Hualong, Yuanben, Kangchen, Sammi

1. Shopee

   • 1.4.0 RC1 VOTE is passed. Yiyang is handling post vote release steps.

2. DiDi

   • Hit one DN data read performance issue after a lot of files are deleted. The performance is caused by the lock contention between data read and block deletion. The jstack shows that all read threads are waiting to acquire container read lock. Filed https://issues.apache.org/jira/browse/HDDS-10146 to continue the investigation and find the right solution.
     One way to alleviate the lock contention is introduced by HDDS-9107, which introduces a property "hdds.datanode.block.deleting.max.lock.holding.time" to define how long the block deletion will hold the container lock in one iterator.
   • DN decommission is slow after most of the containers on the DN are replicated. It's not sure if the DN decommission is stuck or not. The investigation is going on. Suggested to check below two umbrella JIRAs to see if any fixes are related and usable.
     HDDS-8699 Further Replication Manager Improvements
     HDDS-7759 Improve Ozone Replication Manager

3. Qihoo

   • Found one container balancer performance bottleneck. When balancing a DN with 200K containers, it will take around 2~3 hours to select one container candidate to move. The most time consuming part is ContainerBalancerSelectionCriteria#getCandidateContainers. Yiyang has a solution for this, and will submit a PR later. HDDS-10160.
   • Working on container replication grpc zero-copy improvement work, will submit PR later. HDDS-10144