feat: parallel chunk PUT and GET

[sanity]

Problem

ChunkedPack publish and fetch are serial: each chunk operation waits for the previous to complete. Against the Freenet host under load each chunk takes ~60s, so a 200-chunk push (full freenet-core history at 1 MiB chunks) takes ~3-4 hours of wall time. That's the dominant pain point for routine work on big repos and is called out as the last "not yet supported" item in the README.

Approach

Run chunk operations in parallel across a small pool of WebSocket connections to the local Freenet node.

• Open N connections up front (Vec<Arc<Mutex>>).
• Round-robin chunks to connections (chunk_index % N).
• Drive completion with futures::stream::iter(...).map(...).buffer_unordered(N).
• For fetch, slot results into Vec<Option<Vec>> indexed by chunk position so reassembly is in manifest order regardless of completion order.
• Manifest PUT and re-GET stay single-shot on the first connection (they are one-of-one operations).

N defaults to 8 (DEFAULT_PARALLEL_OPS) and can be overridden with FREENET_GIT_PARALLEL_OPS=K. Setting it to 1 forces serial behavior. The pool is capped at the chunk count to avoid wasted connections for small payloads.

API change

publish_chunked_pack and fetch_chunked_pack now take ws_url: &str instead of &mut WebApi because they open their own pool internally. The _with_progress variants gain a parallelism: usize parameter. Progress callbacks fire on completion (with completed-count) instead of before each op, which is the correct semantics for out-of-order parallel work.

This is a breaking lib API change. The lib is internal to the binaries in this same crate; no external consumers known. The callers in git-remote-freenet.rs are updated.

Rejected alternatives

• Multiplex one connection. WebApi is !Sync and the WebSocket stream isn't designed for interleaved txns from one client; one connection per concurrent op is simpler and the resource cost is small.
• Spawn N worker tasks pulling from a channel. Equivalent in throughput but more code; the buffer_unordered form is shorter and the per-task index propagation is direct.

Testing

New unit tests in chunked.rs::tests:

• assemble_chunks_concatenates_in_order: gives the assembler the same chunks in slot order and again in shuffled completion order, asserts both produce the same byte vector. Load-bearing piece for parallel fetch correctness.
• assemble_chunks_errors_on_missing_slot: a None slot must error rather than silently truncate.
• assemble_chunks_errors_on_total_size_mismatch: defends the manifest total_size invariant.
• parallelism_from_env_handles_unset_zero_and_invalid: env-var parsing including 0 and non-integer fallback.

Manual e2e against a real Freenet node is the planned next step before merge; will report results in a follow-up comment.

Why didn't CI catch this?

The serial implementation wasn't a bug, it was a perf limitation. There is no slow-operation regression CI check. The new parallel impl could regress in two ways:

• Reassembly mis-ordering. Closed by the assemble_chunks unit tests.
• Hung locks / deadlock from an .await while holding a mutex guard wrong. Caught by running the existing test suite under tokio runtime; could be made stronger with a mock-WebApi integration test in a follow-up.

Out-of-scope (filed for follow-up if needed)

• Parallelize rescue_chunked in main.rs (different control flow: GET-then-PUT per chunk).
• Streaming chunks from git pack-objects directly without staging the full pack in memory. Bigger refactor; do separately if it's needed.

Fixes

Closes #7

[AI-assisted - Claude]

[sanity]

Review pass complete

Ran four internal review agents (code-first, testing, skeptical, big-picture) and Codex in parallel. Summary of what was applied and what was punted.

Applied to this branch

• [Codex P1] Graceful pool degradation. `open_pool` no longer aborts the whole publish/fetch if the local node refuses connection N+1; it proceeds with however many it got and warns. Falls back to error only if it cannot open even one (matching the pre-parallel single-connection behavior).
• [Testing P1 + Skeptical M1] Race-free env parsing. Split `parallelism_from_env` into a thin reader plus a pure `parse_parallelism(Option<&str>)` helper. Tests now exercise the parser directly instead of mutating process env state. Avoids the known anti-pattern that becomes a hard error in edition 2024.
• [Testing P1] Real out-of-order completion test. Extracted `collect_chunks_into_slots` from the fetch loop and added two `#[tokio::test]` cases that drive it with a deliberately scrambled `stream::iter`. The previous "shuffled" test was testing Vec iteration, not the load-bearing reassembly logic.
• [Skeptical N1] Removed redundant `drop(stream)` calls that were no-ops.
• [Skeptical N3] Unified the actual_parallelism clamp form between publish and fetch.
• [Skeptical N4] Tightened the Arc<Vec> comment about phase 1 vs phase 2 cloning.
• [Skeptical N5] Added a comment that phase 2's BLAKE3 verify is belt-and-braces with wsclient::get_pack's own check.
• [Skeptical N7] Renamed shadowed `completed` to `puts_completed` / `verifies_completed`.
• [Skeptical M2] Documented the uniform-per-chunk-latency assumption: round-robin + one stuck chunk on conn k means siblings on that conn queue serially.
• [Big-picture] Updated the two user-facing progress strings to disclose the parallelism N, so users don't multiply ~60s × 200 chunks in their head.
• [Big-picture] Removed the stale README "parallel chunk uploads not yet shipped" caveat in an earlier commit on this branch.

Filed as follow-ups (out of scope for #7)

• [Skeptical M4 + M5] wsclient: put_contract / get_state can leave stale responses on a WS connection after retry or error #9 — wsclient `put_contract` / `get_state` can leave stale responses on a connection after retry. Pre-existing; feat: parallel chunk PUT and GET #8 increases the surface area, doesn't introduce. Worth fixing in its own PR.
• [Code-first] wsclient::put_pack: take &[u8] instead of Vec<u8> to eliminate per-chunk WASM clone #10 — refactor `wsclient::put_pack` to take `&[u8]` so phase 1 can drop the per-chunk WASM deep-clone. Roughly 1 GB of transient allocations per push for a 200-chunk freenet-core history.
• [Big-picture / out-of-scope from Parallel chunk PUTs for ChunkedPack #7] Parallelize rescue_chunked the same way as publish/fetch #11 — parallelize `rescue_chunked` in `main.rs` the same way.

Validation gap

No CI in this repo, and there is no mock `WebApi`. The orchestration logic (`buffer_unordered` over a `Vec<Arc<Mutex>>` pool, round-robin chunk dispatch, slot-indexed reassembly) is unit-tested in pieces but not end-to-end. Big-picture review noted this and asked for a real-network smoke test before merge confirming 8-way concurrent ops actually work and the speedup is in the expected ballpark. Pending.

[AI-assisted - Claude]