Skip to content

Detach post-receive replication scans onto the F5 drain task (close the #174 inline-park residual) #217

Description

@beardthelion

Problem

The post-receive replication scans run inline in the git_receive_pack handler, so the push HTTP response waits on them. In crates/gitlawb-node/src/api/repos.rs the handler awaits replication_withheld_set (repos.rs:1656), resolve_candidates_for_push (repos.rs:1685), and fail_closed_full_scan_objects (repos.rs:1696) before returning at repos.rs:1957, each parked defer-not-shed on git_encrypt_semaphore.

Two consequences, both shipped as accepted residuals in #174 (see KTD5 in the round-five plan):

  1. Push-visible latency coupling. The park wait is queue-depth-multiplied: once the write permit is released, concurrent tails are not admission-bounded, so under a burst the wait a pusher sees grows with the burst.
  2. Silent durability loss on disconnect. axum drops the handler future when the connection closes, which cancels the in-flight scans. A client or proxy timeout while parked loses that push's IPFS pins and encrypted recovery copies with no error. The F5 coalescing requeue does not cover this, because the park precedes the try_begin gate. There is no reconciliation sweep behind it (see the companion issue), so the loss is permanent until an unrelated later push happens to re-walk the repo.

This is the anti-pattern for durability-critical, client-invisible follow-on work: inline on a request future that is cancelled on disconnect is at-most-once by construction. The established remedy is to hand the work to a detached task that outlives the request.

Fix

Route the push's own replication work through the F5 coalescer's detached drain task instead of running the scans inline. The machinery already exists from #174:

  • EncryptInflight::try_begin (crates/gitlawb-node/src/state.rs) admits a repo with an empty pending slot today (slot.insert(PendingWork::Tips(Vec::new())) at state.rs:323). Seed it with the push's own tip pairs instead, so the just-admitted push's work rides the drain loop from the start.
  • run_encrypt_pin_task + resolve_drain_object_list (repos.rs) already re-resolve announce/withheld/object_list fresh inside the detached task via the same pipeline the handler runs inline. git_receive_pack then returns without acquiring git_encrypt_semaphore at all.

This closes the latency coupling entirely and shrinks the disconnect-loss window from seconds-under-load to the microsecond gap between committing the push to disk and spawning the task.

Open design point (the one real cost): the tail Pinata/gossip/anchor spawn currently reads object_list/announce synchronously off the inline scans (object_list_pinata = object_list at repos.rs:1808, do_pinata_replication = withheld.is_some() at repos.rs:1809). Detaching the scans removes those synchronous inputs, so that second spawn must either take its inputs off the drain task's resolution or re-resolve them itself. Resolve this as part of the change rather than duplicating the withheld/object-list walk.

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    crate:nodegitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorsev:highMajor break or real security/trust risk, no easy workaroundsubsystem:apiNode REST API request/response surfacesubsystem:encryptionEncrypted subtrees, recipient blinding, key zeroizationsubsystem:replicationMirror, replica, and cross-node sync

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions