perf(transfer): own the in-flight FileEntry window in the receiver pipeline - #7229
Conversation
…peline Convert run_pipeline_loop_decoupled from `&'a self` with a borrowed `&'a FileEntry` request vector to `&mut self` with an owned, window-bounded FileEntry clone (cloned as each entry enters the in-flight window). The transfer set is now handed to the loop by flist index; the loop clones `self.file_list[idx]` at push time (O(window) resident), so it no longer holds a borrow into the file list for its whole duration. Behavior-preserving: request generation, the metadata-only itemize merge, flush cadence, and NDX order are unchanged. Verified byte-identical against the prior binary on proto32 and proto29 daemon pulls (identical destination trees and rate-normalized client stdout) with no measurable wall-clock change on an 80k-file pull.
11bdb5a to
f01596f
Compare
|
Verification status — draft pending a later verification pass. Verified: diff reviewed line-by-line. The NOT verified: the byte-neutrality evidence — which is this PR's entire justification — was never produced. A pure borrow-model refactor is only safe if the wire is provably unchanged; CI green proves tests pass, not that. The host run was dispatched but its result is unknown. Required before this leaves draft: proto32 daemon pull AND a proto29 transfer, pre-change vs post-change binaries, destination tree and client stdout diffed (normalize only bytes/sec). If the diff is non-empty, this PR is wrong regardless of CI. Known follow-up: |
Byte-neutrality gate: PASSEDMeasured on aarch64 Linux, default features (so Binaries:
All four transfers exited 0. stdout compared with only This is the evidence the PR was waiting on: a pure borrow-model refactor is only safe if the |
a7db181 to
f01596f
Compare
Pure borrow-model refactor of the receiver pipeline. No behaviour change, no wire change.
run_pipeline_loop_decoupledtook&'a selfand held&'a FileEntryborrows intoself.file_listfor the whole loop - both the inputfiles_to_transferand thein-flight
pending_files_infowindow. While those borrows are liveself.file_listcannot grow, which blocks any future work that needs to append to it mid-loop.
This changes the signature to
&mut self, drops the&'a FileEntryfrom the inputtuple (
Vec<(usize, PathBuf, u32)>), and lets the in-flight window own itsFileEntry- cloned as it enters the window and bounded bypipeline.available_slots(),so the clone count is O(window), not O(total).
What is deliberately unchanged: the
no_transfer_rowsmerge order, the flush andkeepalive cadence, the NDX write order, and the phase-2 redo codec threading
(one
ndx_write_codec/ndx_read_codecpair through both the phase-1 and redocalls - upstream keeps a single connection-wide
prev_positive/prev_negative,io.c:2245, and the redo re-requests through that same state).Verification
cargo fmt --all -- --checkandcargo check --workspace --all-featuresare clean onx86_64 Linux, and
cargo nextest run --workspace --all-featuresruns there against thisbranch.
Byte-neutrality was re-checked against a release binary built from the merge base
(
8c68fdb46) over six daemon-pull scenarios: fresh and delta-seeded pulls on protocol 32and protocol 29, plus an
--append-verify --ignore-timesrun on each protocol that forcesthe phase-2 verification redo. Both sides of each transfer used the binary under test. For
every scenario the destination tree (per-file md5, type, mode, size, mtime, symlink target)
and the normalized client stdout - itemize rows plus the whole
--statsblock, includingTotal bytes sent/Total bytes received- are identical. Only thebytes/secfigure isnormalized.
daemon_pull_forced_verification_failure_recovers_via_redo- the regression guard for theredo NDX codec threading - passes on this branch (1 passed).
Known follow-up, filed separately:
build_files_to_transferstill yields the4-tuple and the caller discards the entry. Changing the producer would have widened
this diff, which is gated on byte-neutrality, so the shim stays for now.