Skip to content

Release v0.16.0

Latest

Choose a tag to compare

@github-actions github-actions released this 18 Nov 02:08
· 8 commits to main since this release

last:n from a per-topic count (#181)

The store keeps a frame count for each last:n topic. An append that takes a topic over n removes the oldest frame in the same commit; 0.15.1 scanned back n index entries on every such append. A topic is counted the first time a last:n frame lands on it, so a 0.15 store opens as is.

per append, a topic held at n by last:n 0.15.1 0.16.0
index entries read, n = 50,000 50,001 1
gc work, n = 50,000 17.66 ms 0.14 us
index entries read, n = 5,000,000 5,000,001 1
gc work, n = 5,000,000 759.3 ms 0.13 us

A topic is trimmed by an append that carries last:n. Appends without it can take the topic past n; the next last:n append brings it back.

POST /append-batch (#158)

By kiil, who also designed batched appends in the store: his Store::append_batch in #158 is the design Store::append_many (#168, 0.15.0) follows, and this route now calls it.

printf '%s\n' \
    '{"topic":"clip.add","meta":{"n":1}}' \
    '{"topic":"clip.add","meta":{"n":2},"ttl":"last:100"}' |
  curl --unix-socket ./store/sock --data-binary @- http://localhost/append-batch

Each NDJSON line is a frame: topic, and optionally meta, ttl and hash (content stored with POST /cas). The frames get consecutive ids and commit together. A malformed line, an invalid topic, or more than 1024 frames returns 400 and stores nothing.

one connection frames/s
POST /append/{topic}, 1 frame per request 29,700
POST /append-batch, 1000 frames per request 397,000

Old CAS content is deleted (#184)

A frame's content lives in the CAS, under its hash. In 0.15.1, CAS content was never deleted, even after its frame was gone. In 0.16.0, a file in the CAS is kept while some frame in the stream has its hash, or for an hour after it was written; after that it is deleted.

CAS content 0.15.1 0.16.0
of a frame still in the stream kept kept
of a removed or trimmed frame kept deleted after an hour
of an ephemeral frame kept deleted after an hour
stored with POST /cas, never appended kept deleted after an hour

Deletion runs when the server starts, and on POST /gc:

$ curl --unix-socket ./store/sock -X POST http://localhost/gc
{"in_use":18,"recent":87,"deleted":28720,"bytes_freed":10700000000}

Breaking

  • Nushell 0.117.0 (#178). Actor, service and action scripts run on it, so its removals reach them: filter (use where), -i on get, select and reject (use -o), and let-env. Code that embeds xs needs nu-* 0.117, and building from source needs Rust 1.97.1.
  • CAS content is deleted an hour after it was written, unless a frame in the stream has its hash (#184). Content you store with POST /cas and want to keep needs a frame that references it.

A store used by 0.16 can't go back to 0.15

0.16 keeps a frame count for each last:n topic in the store (#181). 0.15 doesn't know about the counts, so its appends and trims leave them wrong. If the store then returns to 0.16, a count that reads too high makes it trim frames the topic should keep.

Fixes

  • An idle duplex service waits for its next frame instead of polling every 10 ms (#182).
  • One-shot commands flush stdout before they exit (#179).

Changelog

  • feat: enforce last:n from a per-topic count, in the append's own commit (#181)
  • feat: delete content no frame uses, at server start and on POST /gc (#184)
  • feat: add POST /append-batch, by kiil (#158)
  • fix: a duplex service's input waits for frames instead of polling every 10ms (#182)
  • ci: copy the libproc bindings to the folder rustc names, for Rust 1.100 (#183)
  • chore!: update to nushell 0.117.0 and datastar 1.0.5 (#178)
  • ci: Linux release builds use rust:1-alpine, each on a runner of its own CPU (#180)
  • fix: one-shot commands flush stdout before exiting, so their output is not lost (#179)