botjim v0.13.3

Brings the torrent-style swarm close to the theoretical optimum by fixing the flash-crowd problem — many joiners starting together each pulling almost the whole file off the seed.

HRW first-copy partition

For each piece only the seed holds, exactly one joiner — chosen by rendezvous (HRW) hashing over the peer set — is responsible for pulling the first copy off the seed; every other joiner leaves that piece for its owner and pulls it from the mesh instead. So the seed uploads each piece about once and the joiners exchange the rest.

Measured across real containers (seed uplink capped to 25 Mbit, 50 MB artifact, 4 concurrent joiners):

• seed redundancy 2.9x → 1.54x (1.0x = perfect mesh — each piece uploaded once).
• mesh-on completion 54s → 36.4s. The peer-to-peer speedup over a seed-only swarm is now ~55% (was ~33%), near the ~2.2x ceiling for this seed rate.

This builds on the v0.13.2 fixes (a critical endgame over-fetch bug that made a joiner receive up to 40× the file, plus mesh-first picking and per-joiner shuffle).

Also in this line

Faster piece propagation (100 ms re-announce poll with an O(1) availability counter), and peer selection that tries non-seed holders before the seed.

No wire-format break; interoperates with v0.11+. Signed SHA256SUMS.

botjim v0.13.1

Makes the swarm mesh react quickly to new pieces, and adds container-based mesh verification.

Faster mesh ramp-up

A joiner used to re-announce its catalog only every 15s, so peers learned what it could serve too slowly and the mesh stayed under-used. It now re-announces within ~1s of acquiring new pieces. Measured across real containers (seed uplink capped to 25 Mbit, 50 MB artifact, 4 concurrent joiners), the peer-to-peer speedup over a seed-only swarm grew from ~18% to ~30% (58.6s vs a seed-only 83.5s).

Container test suites (this session)

The repo now ships docker E2E harnesses that were used to verify the release across real container boundaries:

• containers_mesh.sh — a 6-container torrent swarm; the seed is killed mid-run and 4 joiners still complete peer-to-peer.
• containers_auth.sh — token/pass/cloak accept+reject matrix.
• containers_relay.sh — broker e2ee pairing (broker sees only ciphertext).
• containers_netem.sh — encrypted transfer + kill-9 resume under 40ms latency + 1% loss.
• containers_ramp.sh — quantifies the mesh ramp (mesh-on vs mesh-off).

No wire-format break; interoperates with v0.11+. Signed SHA256SUMS.

botjim v0.13.0

Completes the torrent-style swarm with proper piece selection, and folds in the throughput work from the same push.

Swarm piece-picking

• Live rarest-first: each worker fetches the scarcest still-needed chunk based on the current peer set (re-announced every 15s), so a piece that turns rare mid-download is prioritized — no longer fixed at start.
• Endgame: for the tail, a free worker duplicates an outstanding piece across another peer; the first verified copy wins. One slow peer can no longer stall a download at 99%.
• Per-chunk retry: a failed fetch returns that chunk to the queue (a peer may reappear via re-announce) and only fails the whole join after exhausting every source — a transient miss no longer aborts everything.
• The tracker hands each announce a random subset of the room (seeds always included), so a large swarm spreads load instead of everyone learning the same list.

Verified with a 5-node run: the seed is killed, then three joiners complete purely from one seeding peer.

Throughput (same release line)

• Small chunk frames are coalesced into a single write (header+payload, +checksum) — many-small-files +66% in the benchmark.
• --fast no-compression profile, per-chunk checksum dropped on encrypted sessions, per-chunk lock removed from the sender, O(1) resume bookkeeping — from v0.12.

No wire-format break; interoperates with v0.11+. SHA256SUMS is signed; clients verify it on botjim update.

botjim v0.12.0

Real torrent-style mesh build-up, plus throughput optimizations.

Swarm is now a real mesh (not leech-only)

Previously every joiner downloaded only from the seed — a joiner advertised pieces it held but physically could not serve them (the serve path read the finalized file, which does not exist mid-download), and the process exited the instant it finished. Now a joiner serves the pieces it holds while still downloading, reading verified chunks straight from its in-progress part file. Pieces propagate peer-to-peer, so a swarm ramps toward ~N×(seed rate) instead of being capped at the single seed's uplink.

Verified end-to-end: after the seed is killed, a fresh joiner still downloads the whole artifact from a seeding joiner; two joiners download concurrently and trade pieces.

• swarm join --seed keeps a finished node serving (torrent-style seeding) until Ctrl-C.
• Availability is tracked in memory, so re-announcing what a node can serve no longer re-hashes the whole part file from disk every 15s (was crippling for large artifacts).
• A dest lock refuses a second swarm join into the same directory; serve connections are capped so a popular node isn't connection-stormed.

Throughput

• --fast: a no-compression profile for fast/trusted links (~+22% on incompressible data like model weights, where the default zstd wastes a compress attempt then discards it). Pair with --pass for encryption.
• The per-chunk checksum is skipped on --token/--pass sessions (the AEAD record layer already authenticates every byte) — ~15% CPU back on encrypted transfers.
• Removed per-chunk global-lock traffic in the sender (more --parallel streams no longer hurt a single-file transfer) and made resume bookkeeping O(1); small-file transfers skip a redundant per-file sidecar write.

No wire-format break; interoperates with v0.11. SHA256SUMS is signed (SHA256SUMS.sig); v0.11+ clients verify it on botjim update.

botjim v0.11.0

Authentication-layer redesign (auth v2). Closes the three limitations noted in v0.10.x.

--token now encrypts the stream

Previously --token was only an HMAC proof of knowledge — the data that followed was plaintext (a MITM could read/modify everything) unless you also set --pass. As of v0.11, the token drives the X25519 record layer, so a token-only session is fully encrypted and integrity-protected. --token --pass requires the peer to know both.

Handshake binding — no more silent downgrade

The plaintext FSY1 handshake (which carries the negotiated feature bits and flags) is now folded into the record layer's key schedule and confirmation MAC. A man-in-the-middle who strips a feature bit (e.g. the per-chunk checksum) or alters any handshake field produces a different key on each side and the session fails to establish.

Signed self-update

botjim update now verifies a detached ed25519 signature (SHA256SUMS.sig) over SHA256SUMS against a public key embedded in the binary, before trusting any checksum. A signed build refuses a release whose signature is missing or invalid — so a compromised release or a forged TLS certificate cannot push a malicious binary (the attacker would also need the release private key, which never leaves the maintainer's machine).

Compatibility

The v2 auth wire changed: both ends must be v0.11+ when using --token/--pass. Mixed old/new versions refuse cleanly (a new SecureV2 handshake flag; a pre-v0.11 peer rejects it as unknown) — never a silent downgrade or corruption. Plaintext transfers (no token/pass), relay/swarm e2ee, and --cloak are unchanged.

This release's SHA256SUMS is signed (SHA256SUMS.sig); v0.11+ clients verify it automatically on botjim update.

botjim v0.10.1

Security hardening from a second, subagent-driven review of surfaces not covered in v0.10.0.

Privilege / remote-root (receiver)

• xattr whitelist: received xattrs are now limited to the user.* namespace. A malicious sender could previously set security.capability (file capabilities — effectively cap_setuid on an attacker-supplied binary), POSIX ACLs, or trusted.* on a root receiver — these were applied unconditionally. This was a remote-root vector.
• device nodes gated: block/char/FIFO node creation now requires --devices; a sender could otherwise drop a device node (a window onto raw devices) on a root receiver regardless of negotiation.
• --no-suid (new, opt-in on server + pull): strips setuid/setgid from received files — defense for a root receiver against an untrusted sender.

webfs (botjim serve)

• Symlink escape out of the served root (a symlink inside root → /etc/~/.ssh was followed and streamed) is now blocked (EvalSymlinks + re-check).
• Stored XSS / attribute injection in the directory listing (unescaped filenames) fixed with HTML/URL escaping.
• Default --root . now serves correctly (was 404-ing everything); dotfiles and .fs-part/.fs-meta files are neither listed nor served; non-GET/HEAD → 405.

Decoder / resource defenses

• zstd decoder memory capped at ~18 MiB per chunk (was 256 MiB — a small streaming frame could balloon ~16× per worker, an OOM amplifier).
• Chunk-index bound compared unsigned (a ChunkIdx ≥ 2^63 bypassed the range check); Grid math defensive against a crafted ChunkSize==0 (divide-by-zero) and near-MaxInt64 size (overflow) — was a latent remote panic.

selfupdate / discover / walker

• SHA256SUMS parser handles the GNU * marker and spaced names; empty expected checksum is now refused. Trust-model doc corrected (checksum ≠ signature).
• Discovery beacons carrying terminal/ANSI control sequences are dropped (they were printed raw by botjim peers).
• --include now descends directories (*.txt previously matched nothing nested).

No wire-format break; interoperates with v0.10.0.

botjim v0.10.0

Transfer-integrity hardening, mesh/swarm fixes, and a broad race/robustness sweep.

Data-integrity (engine)

• Delta adoption & sidecar-less resume are no longer trusted blindly. Claims now carry per-chunk hashes; the sender re-verifies its own bytes and answers with Commit (all verified) or ClaimResult (verified bitmap). The receiver finalizes only after in-flight corrections land — fixing stale chunks being left in the final file (15–40% corruption under random delta edits).
• Zero-chunk delta now physically zeroes a possibly-stale part instead of assuming a hole.
• Per-chunk crc32c on data frames (negotiated) — raw/incompressible chunks previously had only TCP's checksum on the wire.
• Adopted-file sidecar routed to a distinct path (was clobbering & deleting a legitimate <final>.json sibling).
• Read-only / grown / locked delta targets fall through to a fresh part instead of stalling the transfer; source-changed-during-transfer detection.

Swarm

• Fix: v2 swarm join was completely broken — the embedded chunk catalog was dropped so the spec hash never matched. Single-file seeding rel/serve-root mismatch fixed too.
• File mode bound into the swarm ID and masked on apply (blocks setuid tampering on unsigned specs); joiners re-announce (were expiring from their room mid-download) and discover late peers; HTTP Range-ignore guard.

Mesh config propagation

• Credential-carrying envelope removed from the jail after apply; bootstrap-replay bounded by signed timestamp; endpoint deletion now propagates; publish records its version before emitting (no split-brain); cross-process config lock.

Relay / cloak

• Spool spill file reclaimed mid-stream (was growing to the full transfer size on disk).
• Cloak reader rejects malformed frame lengths (was a one-packet pre-auth crash), enforces client masking/FIN.

Audit

• Torn-tail repair only truncates a real partial tail; mid-file corruption fails closed instead of erasing tamper evidence.

Plus many races/lock-discipline fixes and new regression tests across all of the above.

Known limitation: --token without --pass authenticates access but does not encrypt the stream; use --pass (or a trusted network) for confidentiality/integrity of the channel itself.

botjim v0.9.0

Chunk catalog v2, HTTP swarm source, continuous sync, mesh config propagation, Prometheus metrics.

• Swarm specs now carry a per-chunk SHA-256 catalog: every chunk is verified as it lands, a token-holding peer that serves wrong bytes is banned immediately, and part files resume chunk-by-chunk
• swarm join --http BASE assembles from static HTTP hosting (Range per chunk) — HF/S3/nginx work as sources
• sync push --watch keeps mirroring as the source changes (debounced, delta re-pushes)
• config publish + pinned mesh key: edit the endpoint list on one node and every node converges — a signed, versioned envelope applied automatically on receipt
• server --metrics :9090 exposes Prometheus counters

botjim v0.8.0

Named endpoints, one-shot sync, receipts, pipe mode, ed25519 spec signing, LAN discovery.

• sync push|pull NAME mirrors with per-target policy from ~/.botjim/config.json (endpoints: addr/token/pass/cloak; autosync: include/exclude/delete/dest)
• --receipt JSON proof-of-transfer with the manifest digest; pipe send/cat for tar|nc muscle memory, engine-backed
• swarm keygen + --verify-key: ed25519-signed swarm specs — tampered or re-signed descriptors fail closed
• server --discover + botjim peers: opt-in LAN presence over UDP multicast
• Disk-space guard on the receiver before any bytes move
• Fixed: send --delete never mirrored (the wire bit was not set); 2>/dev/null runs launched the TUI (/dev/null was classified as a terminal)

Docs (EN-first, Korean edition) fully refreshed; man page and shell completions cover every command.

botjim 0.7.0

botjim 0.7.0 — cloak, HTTP bridge, JSON events, mirror mode; 0.6.0 mesh + audit

cloak (--cloak PATH, both sides): the session rides a WebSocket upgrade — to anyone watching it's HTTP. Plain GETs get a decoy page; only the right path upgrades. Plain and cloaked clients share one port.

v0.6.0 (also in this release)

• Swarm mesh complete: joiners serve verified chunks (swarm join --serve) — bytes enter each LAN once; rarest-first chunk ordering
• botjim audit verify|tail — tamper-evident hash-chained journal (--audit)
• ~/.botjim/config.json — flag defaults; command line wins

v0.7.0

• botjim serve [DIR] — HTTP + Range bridge for browsers/curl/HF downloaders
• --json — NDJSON transfer events on stdout
• --delete — mirror mode: remove destination files missing from the source
• botjim completion bash|zsh|fish + botjim man