Where BLite fits, and a reliability-first roadmap

[CaffeinatedCoder]

Over the past days I have filed 54 issues and sent 42 PRs, 22 of them merged. Since #247, most of the issues come from oracle test suites that check the engine against an exact reference instead of hand-written expected values (#250, #259, #263, #272, #277). Doing that work has given me a picture of where BLite is strong, where it could win users, and what currently stands in the way. I'd like to share that picture, explain the test work behind it, and hear what you think. The direction is yours to set; this is a proposal, not a request.

1. What makes BLite different

No other embedded database for .NET I know of offers all of this in one package:

1. Pure managed code, no native binaries. One NuGet package for net10.0 and netstandard2.1, plus WASM storage backends (BLite.Wasm). There are no RID-specific native assets to ship for iOS, Android, Unity, Blazor WASM or ARM boards.
2. Source-generated mappers instead of reflection. This is the right design for the places where reflection fails: iOS (AOT only), Unity IL2CPP, Native AOT and trimmed apps.
3. Documents, vectors, geo and time series in one file. HNSW vector search next to the documents makes local RAG and semantic search possible without a second store or a native extension.
4. Compliance in the engine. AES-256-GCM encryption at rest, the audit sink, [PersonalData], subject export, CDC field masking, secure erase and retention policies. I don't know of another embedded engine that has these. For EU on-prem, healthcare and public-sector software, they save real work.
5. A familiar API. DocumentDbContext, LINQ, OnModelCreating: EF Core users feel at home.

2. Who BLite is for

The niches where these strengths count most, roughly in order:

Niche	Why BLite	Current alternative
Offline-first MAUI, Unity and Blazor WASM apps	No native assets, source generators, one engine everywhere	Realm, whose Device Sync reached end of life in Sept 2025 (the on-device database lives on as community open source); SQLite + an ORM; LiteDB
On-device AI: local RAG, semantic search	Vectors next to documents, one file, managed code	SQLite + sqlite-vec (a native extension), a separate vector store
Regulated edge and desktop software	Encryption, audit, GDPR primitives built in	SQLCipher + hand-written compliance code
Desktop tools and internal apps on LiteDB	A modern, async, ACID document DB with the same mental model	LiteDB 5 (last stable 5.0.21, July 2024; v6 is in prerelease)

And just as useful to say out loud, who it is not for: server OLTP at scale (SQLite, PostgreSQL), analytics (DuckDB), raw key-value throughput (RocksDB, FASTER). A short "When not to use BLite" section in the README would build trust rather than cost users.

A one-line pitch along these lines:

BLite — the managed, AOT-ready document database for offline-first and on-device AI .NET apps, with encryption and GDPR built in.

3. What stands in the way: trust

The features are there. What a team choosing an embedded database needs next is the confidence that the engine keeps its promises. The oracle suites found places where it currently doesn't. Most already have a fix PR:

What the docs promise	What the tests found	Status
"Durable: WAL ensures data safety even in power loss"	The WAL was never fsynced on any commit path; a power loss drops acknowledged commits and can leave documents pointing at freed pages (#260)	PR #280 (WalSyncMode.Full default, Normal opt-in)
AOT_LIMITATIONS.md calls typed CRUD AOT-safe	Typed collections cannot write under Native AOT (#273); suppressed IL warnings resurface in consumer publishes (#275); BLite.Caching throws when trimmed (#276)	PR #278 for #273, smoke test in #277
Encryption at rest	A page allocated by a transaction that never commits cannot be decrypted (#274)	PR #279
Multi-process WAL	Lost commits (#197, #198, #200, #203, #216)	Fixed. Still open: #188, #220, #226
Index-served queries return what LINQ returns	Indexed and non-indexed paths disagree on nulls, ±0, NaN, duplicates, ordering and mixed types (#210, #212, #247–#249, #254–#257); BLQL semantics (#264–#268); dynamic indexes (#269–#271)	Several PRs open, see the epic
"Zero allocation"	True for reads (FindById 7 KB vs LiteDB 48 KB). Writes allocate more than LiteDB (single insert 216 KB vs 57 KB)	Wording
"45x faster single insert than SQLite"	BLite was measured without a WAL sync; SQLite ran with its default synchronous=FULL. With #280's Full mode a commit costs ~4.5 ms on APFS instead of ~0.14 ms	Re-run after #280

None of this is unusual for a young engine with this many features. But a user who hits one of these in production is unlikely to come back, and the comparisons are the first thing a sceptical reader will check.

4. How the gap gets closed: the documentation as the specification

The fixes in October up to #243 were mostly found one bug at a time through issues. The oracle suites change that: every claim in README.md and AOT_LIMITATIONS.md becomes a check against an exact, simpler reference, run over many generated cases, so one test finds a whole class of divergence. The goal is one oracle for every library section the documentation makes a promise about.

The method

Each suite follows the same rules, described in docs/ORACLE_TESTS.md (#250):

• An exact reference, never a hand-written expected value. LINQ-to-Objects over the same documents, an in-memory model with a committed and a pending view, a sorted dictionary in the index's own key order, the files rebuilt at every crash point, or the JIT build of the same program for the Native AOT build.
• A seed for every oracle. A patch to src/ that reintroduces a real defect, usually by reverting a past fix, and that the test must catch. A test that passes with and without the defect is not a guard. CI checks that every oracle has a seed and that every seed still applies; tools/oracle-seeds.cs proves locally that each one is caught.
• A candidate registry. A divergence that is found but not fixed is listed with its family and its issue. It is reported as skipped once the issue is filed, and the row fails as soon as it agrees again, so the fix PR has to remove the entry. The suite stays green, the known bugs stay visible, and the registry cannot outlive a defect.
• Triage against the docs. When BLite and the reference disagree, the order is fixed: is the oracle wrong; do the indexed and non-indexed paths agree with each other but not with the reference (then it is a semantics question, settled against README.md, and either the code or the docs change); or does only the index diverge (then it is a bug).
• Shrunk failures. A failing sequence is shrunk to the shortest one that still fails, which is the repro in the issue.
• Commit trailers. Found-By: and Defect-Class: in CONTRIBUTING.md, so git log shows which finder found what, in which class.

Coverage by library section

Section	Oracle	Reference	Found	Status
LINQ on typed collections	Differential, 331 query shapes, indexed and unindexed, sync and async	LINQ-to-Objects	#247–#249	PR #250
Transactions and CRUD, single- and multi-process	Model-based: random Insert/Update/Delete/Upsert/bulk/Begin/Commit/Rollback/Checkpoint/Backup/Reopen sequences	In-memory model, committed and pending views	–	PR #250
B-tree indexes and page accounting	Model-based, every query primitive, every key shape	Sorted dictionary; pages reachable + free + queued = pages allocated	#255–#258	PR #259
WAL and crash recovery	Fault injection, three crash modes (process crash, torn write, power loss), ~1,900 crash states	The store model's committed state, never older than the last acknowledged commit	#260–#262	PR #263
DynamicCollection and BLQL	Differential (API and BLQL, indexed and unindexed) and model-based (with CreateIndex, DropIndex, Vacuum)	LINQ-to-Objects and the typed collection	#264–#271	PR #272
Native AOT and trimming	Twin build of a consumer app	The JIT build's output, byte for byte; the publish log against AOT_LIMITATIONS.md	#273–#276	PR #277
Encryption at rest	The store model and crash oracles on an encrypted database	The same oracles as unencrypted	New crash shapes of #274	Local, not pushed yet
BSON and source-generated mappers	Round trip of every supported type and edge value	Identity	–	Next
Composite and nested indexes	More rows in the LINQ differential	The same rows unindexed	–	Next
Vector search (HNSW)	Recall test	Exact brute-force k-NN	–	Next
Geospatial (R-tree)	Differential	A linear scan with the same predicate	–	Next
TimeSeries and retention policies	Model-based	The model with the retention rule applied	–	Next
CDC / Watch	Model-based	The model's committed operations in commit order, nothing from rolled-back transactions	–	Next
Key-value store, TTL, BLite.Caching	Model-based	A dictionary with expiry	–	Next
Backup, VACUUM, single ↔ multi-file migration	Model-based (Backup and Vacuum are already steps)	The same contents before and after	–	Partly; migration next
GDPR: secure erase, subject export, CDC masking	Raw-file and differential checks	No bytes of an erased document left in the file; export equal to the model filtered by subject; no [PersonalData] field in a masked event	–	Next

Each new feature then extends a catalogue rather than adding a test: a new LINQ operator is one more query shape, a new storage operation is one more step in the model and one more set of crash points, and a new public API is one more call in the AOT consumer.

What the suites don't see, and what stays with review: targets other than net10.0 (netstandard2.1, Unity IL2CPP, MAUI), real multi-process races beyond the interleavings the model drives, crashes in multi-process mode and inside Backup, torn page writes, performance, and anything without an exact reference, such as API design.

5. Proposal

1. Reliability before features for one release cycle. Land the durability, encryption and AOT fixes and the query-correctness fixes before adding new features. The tracking epic Epic: reliability before features — durability, AOT, query correctness and oracle coverage #288 lists the issues and PRs in a suggested order.
2. The oracle suites in CI. Merge the stack (test: oracle tests for the LINQ provider and the typed collection store #250 → test(oracles): model-based test of BTreeIndex with page accounting #259 → test(oracles): crash consistency of the WAL and page store by fault injection #263 → test(oracles): differential and model-based tests of DynamicCollection #272 and test(aot): Native AOT smoke test, a consumer under JIT and AOT with the publish log as oracle #277) and agree on which suites run in CI and with what time budget. The AOT smoke test needs clang and a publish, so it may fit a nightly job better than every PR.
3. A stability matrix in the README. Mark each feature stable or experimental. Multi-process mode would stay experimental until Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188, Multi-process WAL: a permanently unreadable record inside the published range stalls replay, checkpoints and truncation #220 and Multi-process follow-ups after #221 (open lock): index fill under the lock, dictionary reload for readers, stale free-list head, ... #226 close. A feature becomes stable when its oracle is merged and its registry is empty.
4. Claims backed by tests. Reword the durability line (fix(wal): sync the WAL to disk before a commit is acknowledged #280 already does) and "zero allocation". Re-run BENCHMARKS.md with the sync mode stated: BLite Full vs SQLite WAL + synchronous=FULL, and BLite Normal vs SQLite WAL + NORMAL.
5. A public testing page. SQLite's reputation rests largely on its testing page. A short TESTING.md, linked from the README, summarising docs/ORACLE_TESTS.md and the coverage table above, would show users what is checked and how.
6. Sharpen the README around the niches above. Lead with the use cases and the pitch rather than the feature list.

6. Questions

• Do the niches match how you see BLite? Is there a use case you have in mind that I've missed?
• Would you accept new oracles per library section in the same form (test, seed, candidate registry), and the Found-By: / Defect-Class: trailers?
• Which suites would you want in CI, and with what time budget?
• Would you accept a stability matrix with experimental labels on some features, tied to oracle coverage?
• Should the durability fix (fix(wal): sync the WAL to disk before a commit is acknowledged #280) and the AOT fix (fix(generator): emit the mapper's BSON keys so typed writes work under Native AOT #278) go out together in the next release?
• Would you accept TESTING.md and the README/BENCHMARKS changes as separate small PRs once the fixes land?

I'm happy to do the docs and benchmark work, and to keep working through the epic and the remaining sections.

[CaffeinatedCoder]

Withdrawn by the author on 2026-10-19; the reasoning is in the comments on #288 and #296. The discussion is still pinned and I can no longer unpin it myself. Could you unpin it when convenient? Nothing else is needed, and it can stay closed as a record.