Skip to content

v0.1.11: parallel chunked PUT and GET

Choose a tag to compare

@sanity sanity released this 01 May 15:08
· 48 commits to main since this release

Highlights

ChunkedPack publish and fetch now run chunk operations in parallel across a small pool of WebSocket connections to your local Freenet node. Default 8 connections, override with `FREENET_GIT_PARALLEL_OPS=N`.

For a 200-chunk push at ~60s per chunk under network load: serial was ~3-4 hours, 8-way is ~25 minutes.

Changes

  • Round-robin chunk-to-connection dispatch with `buffer_unordered`. The first N futures map 1:1 to N connections; subsequent chunks pull from the same conn only after it's released. No inter-chunk contention in steady state.
  • Slot-indexed reassembly so completion order does not affect output bytes.
  • Manifest PUT and re-GET stay single-shot on the first connection.
  • Pool opens lazily and degrades gracefully: if the local node refuses connection N+1 (e.g. it caps per-client connections), we proceed with however many we got. Only fails if we cannot open even one connection.
  • User-visible progress strings now disclose the parallelism N.

API change (lib consumers)

`publish_chunked_pack` and `fetch_chunked_pack` now take `ws_url: &str` instead of `&mut WebApi` because they open their own pool internally. The `_with_progress` variants gain a `parallelism: usize` parameter. Lib API consumers are internal to the crate; no external consumers known.

Install

```sh
cargo install freenet-git
```

Live demo URLs

```sh
git clone freenet::AaRxPZVdWrPh/freenet-core
git clone freenet::2pyvKxrozxgT/freenet-stdlib
```

Closed

  • #7 Parallel chunk PUTs for ChunkedPack

Filed for follow-up

  • #9 wsclient retry can leave stale responses on a connection (pre-existing, increased surface area)
  • #10 `put_pack` should take `&[u8]` to drop the per-chunk WASM clone (~1 GB transient allocs per 200-chunk push)
  • #11 parallelize `rescue_chunked` the same way

[AI-assisted - Claude]