Release v0.14.0

Breaking changes

• A following read now ends when it falls behind (#160). xs holds 4096 live frames for a reader. A reader that got further behind than that used to lose the frames in between, and got a log line it could not see. It saw a stream with a hole in it and no way to know. The read now ends instead. To carry on, read again with after set to the last id you received.

• Frames with a ttl: time:<ms>, written before this release, stay on disk forever. They still vanish from reads when they expire. But the new sweeper, below, works from an index that only newer frames have, so nothing deletes them. Export the store and import it again to get the space back.

• .id unpack returns a different shape (#149). ts_ms replaces timestamp. It is an integer in milliseconds, and pack reads it back exactly. A new when field shows the same time as a datetime, which pack ignores. Update scripts that read timestamp.

• xs::nu::ReadMode is gone, and Store::read is now synchronous (#146). If you embed xs, remove the .await and the mode argument.

• xs needs Rust 1.96.1 and uses Nushell 0.116.0 (#159). Nushell deprecated get -i and reject -i. Use -o instead (#162). @kiil raised both, in #153 and #152.

Highlights

• xs now deletes expired frames from disk. A frame with a ttl: time:<ms> expires that many milliseconds after it is written. Before, a read hid an expired frame and queued it for deletion, so a frame that nobody read was never deleted. A sweeper now runs every second and deletes up to 4096 frames each pass. Set the interval with xs serve --ttl-sweep.

• Retention costs a fraction of what it did. Enforcing ttl: last:<n> used to scan a topic once for every append waiting in the queue, and start each scan from the beginning. It now scans once per batch, resumes where the last scan stopped, and deletes a frame from what the index already told it. A burst of 20,000 appends to a topic under ttl: last:1000 took 4,190 ms. It now takes 11 ms.

• xs cat is 20 times faster (#155, @kiil). It wrote and flushed each frame on its own. It now fills a buffer of up to 256 KiB and writes once. A 500k frame store took 7.3 s. It now takes 0.35 s, which is what curl costs against the same socket. --follow still sends each frame as it arrives.

• Appends without a body are 3 times faster (#157, @kiil). Every append opened a CAS writer before it knew whether a body followed, and opening one creates a temp file. It now opens the writer on the first chunk of the body. One connection did 9,900 appends a second. It now does 34,600.

• GET /metrics reports counters. You get fjall's cache and filter counters for each keyspace, plus the store's own: how many appends it took, how many frames a read scanned against how many it returned, and what retention cost. Every counter is cumulative, so read it twice and subtract to get a rate.

• xs serve --fsync sets the fsync policy. It defaults to interval:1000.

• A service can restart when its pipeline finishes cleanly. restart: "on-success" is the default. Use "never" to stop after one run.

Bug fixes

• read_sync collected every frame before it returned the first one. On a 10M frame store that held about 1.7 GB, and it peaked near 3.4 GB while the vector grew. It now streams.
• fjall ran with one worker thread. One thread starves its flushes under a steady write load, so writes stopped reaching disk. It now runs with two.
• Re-importing a frame under a new topic left the old index entry in place, so the frame answered to both topics. Changing its ttl left the old expiry entry behind (#161).
• A scru128 id lost precision in pack, so unpack | pack did not return the same id (#147).
• Each .cat and .last leaked a tokio runtime (#146).
• A topic read with a from or after bound scanned the topic from the start. It now seeks to the bound.
• xs.stopping now runs on SIGTERM and on a Windows console close. Before, it ran only on ctrl-c.
• A new engine no longer inherits XS_APPEND_META. Only xs sets that variable.
• The Discord example stopped its reconnect loop after a failed identify.

Changelog

• test: poll until .tmp-spawn's server accepts, fixing a CI startup race
• fix: halt discord reconnect loop after failed identify; add std/random import
• docs: prefer committing and pushing after each stable change
• fix: stop leaking a tokio runtime per .cat/.last, consolidate read commands (#146)
• fix: round scru128 timestamp on pack to stop id roundtrip loss
• test: stop test_topics_filter_pulse_still_ticks flaking on interleaved pulse ticks (#148)
• refactor: carry scru128 timestamp as integer milliseconds (#149)
• fix: run xs.stopping on SIGTERM and Windows console-close, not just ctrl-c (#150)
• feat: add a configurable fsync policy, default interval:1000
• feat: add a restart option to services, on-success by default, never to stop after one run
• fix: strip an inherited XS_APPEND_META from a new engine, only xs sets it
• perf: batch gc removals into one write batch per drain
• feat!: sweep expired time ttl frames in the background
• docs: drop the pre-sweeper aside from the ttl page
• fix: keep following past a lagged broadcast receiver
• feat: bump nushell to 0.115.1, rust-version to 1.95.0
• feat: bump iroh to 1.1.0, fix a connection-lifetime bug it exposed
• test: make iroh tests hermetic and kill the client on timeout
• feat: bump bon, console, webpki-roots, dirs, base64, nix, duct
• chore: cargo update, fix deprecated scru128 from_fields
• fix: make read_sync forward iteration lazy
• fix: raise fjall worker threads from 1 to 2
• perf: seek topic reads to the from/after bound instead of scanning the topic
• perf: carry each frame's expiry in its idx_topic value
• perf: resume the expiry sweep and the last:n trim where they stopped
• docs: describe ttl cleanup as asynchronous and batched
• perf: scan a last:n topic once per drain, not once per task
• perf: trim removes frames straight off the index, with no point read
• feat: expose store and fjall keyspace counters at GET /metrics
• perf: widen the idx_topic restart interval to 32
• fix: type .last's topic as any so it runs on nushell 0.115
• feat: bump nushell to 0.116.0, rust-version to 1.96.1 (#159)
• fix: move index entries when a restore changes a frame's topic or ttl (#161)
• fix: replace deprecated get -i and reject -i with -o (#162)
• fix: end a following read that falls behind the live buffer (#160)
• perf: coalesce stdout writes in xs cat (#155)
• perf: open the CAS writer only when an append has a body (#157)