[perf] Refactor MooncakeStore backend with zero-copy upsert API

[0oshowero0]

1. Switch to batch_upsert_from & batch_get_into for tensor
2. Switch to upsert_batch & get_batch for non-tensor
3. Use batch_remove API for data clearning
4. Set hard-pin flag during data writting
5. Use multi-thread to optimize data preparation & transfer workflow

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Refactors the MooncakeStore backend integration to use newer “zero-copy” batch upsert/get APIs, adds buffer-prep utilities, enables hard-pin replication, and parallelizes preparation/transfer work.

Changes:

• Add tensor buffer utilities (grouped allocation, byte sizing, contiguous-region merging).
• Update MooncakeStore client to use batch_upsert_from/batch_get_into for tensors and upsert_batch/get_batch for non-tensors, with multi-threaded batching.
• Switch deletion to batch_remove and hard-pin data during writes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 18 comments.

File	Description
transfer_queue/utils/tensor_utils.py	New utilities for zero-copy buffer preparation and memory-region merging.
transfer_queue/storage/clients/mooncake_client.py	Uses new MooncakeStore APIs, adds hard-pin replicate config, and multi-threaded put/get.
tests/e2e/test_kv_interface_e2e.py	Removes outdated comment about MooncakeStore update behavior.
pyproject.toml	Bumps mooncake-transfer-engine dependency version.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Guide

@ji-huazhong , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
6f0fc065 Merge branch 'main' into mooncak...	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

[ascend-robot]

CLA Signature Pass

ji-huazhong, thanks for your pull request. All authors of the commits have signed the CLA. 👍