Skip to content

v1.0.0 — Fjall 3

Latest

Choose a tag to compare

@anidotnet anidotnet released this 31 Aug 19:05

Nitrite for Rust moves to Fjall 3 — and with it, to a new on-disk format.

Warning

A database written by 0.10.x or earlier cannot be opened by 1.0.0. The break is inside the storage engine, below any layer Nitrite could migrate. Recreate the database from your source of truth, or export before upgrading and import after. See Schema Migration.

Why 1.0.0 and not 0.11.0

The storage engine underneath the adapter changed major version, and the on-disk format went with it. A format break is the loudest thing a release can do to someone already running the library, and it should not arrive behind a version number that reads like a routine step. The API is also settled enough to say so — this release removes the last knob that existed only to serve Fjall 2's garbage collector, and nothing else is waiting to be pulled out.

Performance

Measured on the repo's criterion suite, 0.10.0 as the baseline, two independent runs per side. The in-memory store is untouched by this release and acts as a control: its median reported change is −0.8%, which is where the noise floor sits.

fjall-backed workload median change
CRUD writes (insert single + batch) −94.9%
Full scan (non-indexed search) −94.1%
Spatial queries and index build −92%
Indexed search −91.7%
Concurrent read + mixed (1k docs) −87.6%
Unique index build −71.1%
Full-text index build / query −53.3% / −48.5%
Concurrent insert (2/4/8 threads) −47.2%
All 34 fjall benchmarks −80.2%

The mechanism is the journal, not micro-optimisation. For the same 10,000-message workload holding ~8 MiB of logical data:

0.10.0 (Fjall 2) 1.0.0 (Fjall 3)
journal peak, during the bulk write 1,664 MiB 128 MiB
total after compact() 40.7 MiB 76.1 MiB

Fjall 2 pinned mostly-empty, preallocated 32 MiB journal segments that no partition had flushed past. Fjall 3 writes 128 MiB for the same work, and that missing I/O is most of the speedup. The settled footprint goes the other way — peak transient down 13×, steady state up 1.9× — and both sit far under the suite's 250 MiB-per-10k-messages gate.

One regression: creating a keyspace costs roughly twice as much (40 → 97 ms on SSD, 91 → 148 ms on an external volume). That is a one-time cost per collection and per index, not a per-operation one. Steady-state transaction cost is unchanged. It is also why Transaction/* in the criterion suite reads +151%: that benchmark creates a fresh database and first touches its collection inside the timed region, so every iteration pays exactly one keyspace creation.

Breaking changes

  • On-disk format — see the warning above.
  • compaction_strategy(...) takes nitrite_fjall_adapter::Strategy (Leveled or Fifo) instead of fjall::compaction::Strategy. Fjall 3 replaced that enum with a trait object and dropped size-tiered compaction, so SizeTiered is gone.
  • space_amp_factor(...) was removed. It only ever fed Fjall 2's gc_with_space_amp_target; Fjall 3 folds blob reclamation into compaction and has no space-amplification target to aim at.
  • staleness_threshold(...) still drives blob reclamation but is applied at keyspace creation rather than per garbage-collection call.
  • max_journaling_size(...) now has a 64 MiB floor, up from 24 MiB — Fjall panics below it. The default is 512 MiB.
  • flush_workers / compaction_workers are both still accepted, but Fjall 3 has one shared worker pool, so the adapter sizes it at the larger of the two.

Two things the port had to replace, not just rename

  • The periodic fsync timer. Fjall 3 removed Config::fsync_ms and the thread behind it. Left alone, that would have silently turned the default Durability::Periodic from "durable within fsync_frequency" into "durable only on a clean close" — an unbounded power-loss window. FjallStore runs that timer itself now, waiting on a condvar to a deadline so a spurious wakeup cannot widen the window and close() wakes it immediately.
  • A map handle that lived too long. A Fjall 3 keyspace handle carries a clone of its database, and Fjall 3 holds an exclusive lock over the database directory. Nitrite's core keeps a NitriteMap for every collection and index, so those handles outlived store.close() and a reopen of the same path failed with Locked. FjallMap now releases its handle in close() and dispose(). Fjall 2 had no lock file, so the same over-long lifetime was invisible there.

Dependencies

Crate From To
fjall 2.6.3 3.1.10
tantivy 0.25.0 0.26.1
cargo_toml 0.22.1 1.0.0
rand 0.8.5 0.8.8
thiserror 2.0.17 2.0.20
tempfile 3.23.0 3.27.0

Tantivy 0.26 split TopDocs from the collector it builds, so the FTS search now asks for TopDocs::with_limit(n).order_by_score() — the ranking TopDocs used to imply, so results are unchanged.


Full detail in CHANGELOG.md.