[Store] Add dynamic hot replica fanout

[zxpdemonio]

Description

RFC: #3388

This PR adds the first-stage dynamic hot replication path for Mooncake Store.

The target problems are hot objects becoming single-node bottlenecks and repeated remote-domain reads paying avoidable access cost. This PR focuses on the baseline dynamic multi-replica fanout path for immutable MEMORY objects. Cross-domain placement fields remain reserved for the next phase; non-empty requester/target domain hints are rejected in this first-stage PR so the baseline path cannot accidentally claim domain-aware placement.

The implementation uses centralized admission with distributed execution. Master records a bounded per-key heat window from the Get / BatchGet path via GetReplicaList / BatchGetReplicaList. When the heat threshold is crossed in enforce mode, the read path only queues a lightweight admission trigger. A Master background worker validates metadata, object version, replica limits, in-flight state, placement, and lease validity before creating an asynchronous copy task. The source-side CopyStart path reserves quota and now rolls back dynamic pending state if that reservation fails.

Copy execution is source-side asynchronous in this PR. Master creates the existing REPLICA_COPY task for the selected source client, and the Store task worker drives CopyStart -> transfer -> CopyEnd. Reader-local promotion is intentionally left out of this PR because foreground read latency has higher priority, and that path needs a separate async buffer-lifetime design.

The PR includes:

• internal dynamic replica proposal and lease model for Mooncake-internal expansion control;
• observe/enforce configuration and Master-side heat-window admission;
• automatic enforce-mode expansion from the Get / BatchGet path through a lightweight background admission queue;
• max dynamic replica limits;
• preferred-target placement, with requester/target domain fields reserved but rejected until domain-aware placement lands;
• lease/version-fenced dynamic CopyStart, CopyEnd, and CopyRevoke validation;
• dynamic copy task publication only after Master pending lease state is installed, with rollback on task-submit failure;
• source-client ownership validation for dynamic CopyStart;
• immediate dynamic pending cleanup when dynamic CopyStart fails before task creation;
• expired dynamic pending/lease cleanup from the existing task-cleanup worker, including failure of unconsumed pending copy tasks;
• fenced durable cleanup for expired in-flight copy tasks so delayed HA callbacks cannot remove newer task state;
• source selection with stable tie-breaking across existing readable MEMORY replicas;
• target selection with existing-replica host anti-affinity, utilization scoring, and stable tie-breaking;
• eviction/reclaim-driven shrink semantics with dynamic replica metadata cleanup and recreate cooldown;
• bounded heat-window cleanup when the window reaches its entry limit, avoiding full scans in the read path; and
• focused tests for admission, idempotency, observe mode, async Get/BatchGet enforce triggers, replica limits, source/target selection, stale task fencing, reaper cleanup, source-client validation, failed-start cleanup, expired-pending isolation, short proposal deadline clamping, first-stage rejection of domain hints, pending-task retirement, and copy lifecycle validation.

This PR deliberately excludes TCP transport optimization, RL weight transport, external framework integration, reader-local promotion, a separate shrink controller, active reclaim policy changes, and a full cross-domain cost model.

Shrink is intentionally delegated to the existing memory eviction/reclaim path. When eviction removes a dynamic replica, Master forgets the dynamic replica record and records a short recreate cooldown so hot-read admission does not immediately re-add the capacity that reclaim just freed.

Rollout note: dynamic enforce mode should be enabled only after both Master and Store workers include this change. Mixed deployments where old Store workers consume dynamic REPLICA_COPY payloads are not supported by this first-stage PR.

Replica Selection Note

Dynamic replication creates additional replicas; client-side read selection is still handled by the existing replica selection path. For remote MEMORY replicas, Mooncake keeps the first remote replica unless MC_STORE_REPLICA_SCORING=1 is set or a scorer is injected. The built-in scorer prefers RDMA over TCP, but equal-score replicas still keep Master return order, so it is not equal-cost load spreading.

For tests or deployments that depend on remote-replica preference, enable MC_STORE_REPLICA_SCORING=1 together with dynamic replication. True same-tier fanout still requires local placement, an injected load/hash-aware scorer, or a follow-up same-tier spreading policy.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

./scripts/code_format.sh
cmake --build build-dynamic-shrink --target dynamic_replication_test task_executor_test task_manager_test -j32
ctest --test-dir build-dynamic-shrink -R 'dynamic_replication_test|task_executor_test|task_manager_test' --output-on-failure
git diff --check

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done: dynamic_replication_test, task_executor_test, task_manager_test, format, and git diff --check passed.

dynamic_replication_test now has 27 cases covering accepted Master-side hot admission, async enforce-mode Get and BatchGet path auto expansion, observe-mode dry run behavior, max replica suppression, below-threshold rejection, proposal idempotency, conflicting proposal-id reuse, short proposal deadline clamping, first-stage rejection of domain hints, copy lifecycle completion, invalid-target cleanup, bounded heat-window cleanup, stale dynamic task rejection, non-source dynamic CopyStart rejection, failed dynamic CopyStart pending cleanup, stale task isolation from newer pending state, expired dynamic copy cleanup, expired dynamic pending isolation from ordinary Copy, version/lease fencing for dynamic CopyStart / CopyEnd / CopyRevoke, eviction-driven dynamic replica shrink cooldown, source stable tie-breaking, and target host anti-affinity. task_manager_test now covers retiring a pending task without later assigning it to a worker.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit run --all-files and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

Touched C++ files build in dynamic_replication_test, task_executor_test, and task_manager_test. Full pre-commit has not been run in this pass.

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

OpenAI Codex helped research the design, draft the RFC, implement the initial proposal/lease flow, run focused tests, and perform self-review. The human submitter is responsible for reviewing and defending the change before merge.

[ykwd]

Thanks for the PR.

One question on the external usage path: since Master only accepts SubmitReplicaActionProposal and does not auto-propose from GetReplicaList (that heat observation is observe-only), could you clarify the expected caller scenario and end-to-end flow for external users?

Specifically:

Who is expected to observe heat and submit proposals in production (domain-local controller, Store client, sidecar, something else)?
What is the recommended sequence after a lease is returned — who picks up the REPLICA_COPY task and drives CopyStart / CopyEnd?

[zxpdemonio]

Thanks for raising this. We considered this because we do not want Master to become the hot-path decision maker for every read. The intended production flow is closed inside Mooncake, with the reader-side Store client path observing heat and proposing promotion; Master only arbitrates and fences the action.

The PR now supports two internal execution paths:

1. Source-side async copy, which is the baseline. The reader-side path observes heat and submits SubmitReplicaActionProposal. Master validates the proposal, selects source/destination, grants a lease, and creates the existing REPLICA_COPY task for the selected source client. The source task worker then drives CopyStart -> transfer -> CopyEnd.

2. Reader-local promotion, used only when the reader can safely reuse the full-object read buffer. The proposal must set allow_reader_local_promotion, include requester_client_id, and use a preferred_target_segment owned by that requester. In that case Master returns a lease without a source-side task, and the reader-side path asynchronously drives CopyStart -> local target write -> CopyEnd using the bytes it already read. CopyStart rechecks that the requester still owns the target segment, so the reader does not bypass Master fencing.

sequenceDiagram
    participant R as Reader Store client
    participant M as Master
    participant S as Source Store client
    participant D as Target segment

    R->>M: GetReplicaList / read object
    R->>R: observe heat and pick local preferred target when available
    R->>M: SubmitReplicaActionProposal(requester_client_id, preferred_target_segment, allow_reader_local_promotion)
    M->>M: validate version, quota, source, target, lease
    alt reader-local target is owned by requester
        M-->>R: lease(reader_local_promotion=true, task_id=0)
        R->>M: CopyStart(source, target)
        M->>M: recheck requester owns target and stage metadata
        R->>D: async write from read buffer
        R->>M: CopyEnd
    else source-side fallback
        M->>S: enqueue REPLICA_COPY task
        M-->>R: lease(reader_local_promotion=false, task_id)
        S->>M: CopyStart(source, target)
        S->>D: transfer object
        S->>M: CopyEnd
    end

Loading

So the answer to “who observes heat and submits proposals” is: the Mooncake reader-side Store client path. Master remains the consistency/lease authority, but it does not auto-propose from GetReplicaList, which avoids putting distributed heat aggregation and per-read expansion decisions on Master.

For “who picks up the task”: in the source-side baseline, the selected source client picks up the existing REPLICA_COPY task. In the reader-local promotion path, there is intentionally no source-side task; the reader owns the asynchronous promotion because it already has the bytes and the target segment is local to it.

Could you take a look at whether this split matches the direction you expect? I think keeping both paths in this PR is useful: source-side copy is the conservative fallback, while reader-local promotion avoids duplicate transfer when the reader has already paid the read cost, and both paths still use the same CopyStart/CopyEnd metadata fence.

[ykwd]

Thanks for the write-up. I am still a little bit confused about the following issues:

1. Who is supposed to submit the replica-expansion proposal?

My reading is that this is meant to stay inside Mooncake: the reader-side Store client observes heat and calls SubmitReplicaActionProposal, and Master only arbitrates / fences. It is not intended to be driven by an external third-party scheduler.

Could you confirm that? I ask because the production loop is not wired on the Store Get path yet — there is an RPC and Master-side lease/task logic, but the client does not currently count heat or submit proposals after a read. If the proposer is really the internal Store client, that closed loop still needs to live in Mooncake. If an external scheduler is in scope, that should be stated explicitly, because it changes who is allowed to set access_frequency_qps / hits / preferred_target_segment.

2. Why should the client make the “this key is hot enough to expand” decision?

A Store client can only see its own access frequency for a key. That seems like a weak signal for replica-count expansion:

• If a key is genuinely hot, it is hot across the cluster. Looking at one client’s local window can miss that, because the accesses may be spread over many readers and never trip any single client’s threshold.
• The reverse is also true. If sessions are sticky and the same client keeps seeing the same key, that client will observe high local QPS even when the rest of the cluster almost never reads it. Expanding a MEMORY replica in that case looks like a local cache preference, not a cluster-wide replication decision.

So I am not sure why heat admission belongs on the client. If the concern is Master becoming the hot-path decision maker on every read, that is understandable — but the current code already pays most of that cost. In observe mode, GetReplicaList already updates Master-side per-key windows (ObserveDynamicReplicationAccess). Once that counter is updated, deciding “crossed threshold → admit expansion” is a cheap local check, not a new distributed aggregation problem.

In enforce mode Master currently does not use that window. It trusts the proposer-supplied access_frequency_qps / hits and then does placement, lease, and fencing. That means we still have Master-side heat machinery, but the actual admit/deny decision is based on a local (and currently unimplemented) client view that Master cannot verify.

[zxpdemonio]

Thanks for the questions. I agree the previous description was not clear enough and may have made the reader-side path look like the hotness decision maker. Let me separate the two parts.

1. External usage path

For this PR, the dynamic replication loop is intended to be closed inside Mooncake Store. We are not introducing an external scheduler or framework integration contract in this phase.

SubmitReplicaActionProposal is an internal control-plane entry used by Mooncake Store for dynamic replica expansion. If upper-layer frameworks need to participate later, I think that should be handled in a separate phase with a clearer external contract. I will update the RFC / PR wording to avoid implying that this PR already supports that external integration path.

2. Hotness admission and copy execution

The intended model is centralized admission with distributed execution.

The reader-side Store client is only a trigger. Master is the admission authority because it owns the aggregated read-observation signal from the Get / BatchGet path, via GetReplicaList / BatchGetReplicaList. A single reader should not be treated as the authority for object hotness.

To avoid turning Master into a heavy execution bottleneck, Master only does lightweight admission and control-plane work: maintain a bounded per-key heat window, check threshold / pending / cooldown / replica limit / version / placement, then grant a lease and enqueue a copy task. The actual data movement remains distributed and asynchronous: the selected source Store worker picks up REPLICA_COPY and drives CopyStart -> transfer -> CopyEnd.

I am also removing reader-local promotion from this PR. Preserving foreground read latency is the highest priority, and reader-local promotion needs a more careful async buffer-lifetime design. This PR will keep only the source-side async copy path.

sequenceDiagram
    participant Reader as "Reader / Store client"
    participant Master as "Mooncake Master"
    participant Source as "Source Store worker"
    participant Target as "Target segment"

    Reader->>Master: "Get / BatchGet"
    Master->>Master: "Update per-key heat window"
    Master-->>Reader: "Replica list"

    alt "Heat threshold not reached"
        Master->>Master: "No expansion"
    else "Heat threshold reached"
        Master->>Master: "Admission: pending / cooldown / limit / version / placement"
        Master->>Master: "Select source replica and target segment"
        Master->>Source: "Queue REPLICA_COPY task"
        Source->>Master: "FetchTasks"
        Source->>Master: "CopyStart"
        Source->>Target: "Async copy"
        Source->>Master: "CopyEnd"
        Master->>Master: "Mark dynamic replica readable"
    end

Loading

[ykwd]

Thanks for the explanation. I understand now.

[Icedcoco]

I re-reviewed the latest head (23357e8d). The regular CopyStart compatibility regression from the previous revision has been fixed by restricting the zero-target early return to dynamic copies.

I verified the focused MasterServiceTest.CopyStart case and reran the complete master_service_test, together with the dynamic replication, tenant quota, task manager, and task executor tests. All passed locally, and the current CI checks are green.

No blocking issues found. LGTM.