Repository navigation
last:n from a per-topic count (#181)
The store keeps a frame count for each last:n topic. An append that takes a topic over n removes the oldest frame in the same commit; 0.15.1 scanned back n index entries on every such append. A topic is counted the first time a last:n frame lands on it, so a 0.15 store opens as is.
per append, a topic held at n by last:n |
0.15.1 | 0.16.0 |
|---|---|---|
index entries read, n = 50,000 |
50,001 | 1 |
gc work, n = 50,000 |
17.66 ms | 0.14 us |
index entries read, n = 5,000,000 |
5,000,001 | 1 |
gc work, n = 5,000,000 |
759.3 ms | 0.13 us |
A topic is trimmed by an append that carries last:n. Appends without it can take the topic past n; the next last:n append brings it back.
POST /append-batch (#158)
By kiil, who also designed batched appends in the store: his Store::append_batch in #158 is the design Store::append_many (#168, 0.15.0) follows, and this route now calls it.
printf '%s\n' \
'{"topic":"clip.add","meta":{"n":1}}' \
'{"topic":"clip.add","meta":{"n":2},"ttl":"last:100"}' |
curl --unix-socket ./store/sock --data-binary @- http://localhost/append-batchEach NDJSON line is a frame: topic, and optionally meta, ttl and hash (content stored with POST /cas). The frames get consecutive ids and commit together. A malformed line, an invalid topic, or more than 1024 frames returns 400 and stores nothing.
| one connection | frames/s |
|---|---|
POST /append/{topic}, 1 frame per request |
29,700 |
POST /append-batch, 1000 frames per request |
397,000 |
Old CAS content is deleted (#184)
A frame's content lives in the CAS, under its hash. In 0.15.1, CAS content was never deleted, even after its frame was gone. In 0.16.0, a file in the CAS is kept while some frame in the stream has its hash, or for an hour after it was written; after that it is deleted.
| CAS content | 0.15.1 | 0.16.0 |
|---|---|---|
| of a frame still in the stream | kept | kept |
| of a removed or trimmed frame | kept | deleted after an hour |
| of an ephemeral frame | kept | deleted after an hour |
stored with POST /cas, never appended |
kept | deleted after an hour |
Deletion runs when the server starts, and on POST /gc:
$ curl --unix-socket ./store/sock -X POST http://localhost/gc
{"in_use":18,"recent":87,"deleted":28720,"bytes_freed":10700000000}Breaking
- Nushell 0.117.0 (#178). Actor, service and action scripts run on it, so its removals reach them:
filter(usewhere),-ionget,selectandreject(use-o), andlet-env. Code that embeds xs needsnu-*0.117, and building from source needs Rust 1.97.1. - CAS content is deleted an hour after it was written, unless a frame in the stream has its hash (#184). Content you store with
POST /casand want to keep needs a frame that references it.
A store used by 0.16 can't go back to 0.15
0.16 keeps a frame count for each last:n topic in the store (#181). 0.15 doesn't know about the counts, so its appends and trims leave them wrong. If the store then returns to 0.16, a count that reads too high makes it trim frames the topic should keep.
Fixes
- An idle duplex service waits for its next frame instead of polling every 10 ms (#182).
- One-shot commands flush stdout before they exit (#179).
Changelog
- feat: enforce last:n from a per-topic count, in the append's own commit (#181)
- feat: delete content no frame uses, at server start and on POST /gc (#184)
- feat: add POST /append-batch, by kiil (#158)
- fix: a duplex service's input waits for frames instead of polling every 10ms (#182)
- ci: copy the libproc bindings to the folder rustc names, for Rust 1.100 (#183)
- chore!: update to nushell 0.117.0 and datastar 1.0.5 (#178)
- ci: Linux release builds use rust:1-alpine, each on a runner of its own CPU (#180)
- fix: one-shot commands flush stdout before exiting, so their output is not lost (#179)