v6 architecture: twelve decisions to make before the format breaks

[CaffeinatedCoder]

#296 tracks what v6 needs around the code (migration guide, docs, release engineering). This Discussion is about the code itself. While reviewing it for #296 I found that the defects of the last weeks cluster in a few structural places, and that v6, which breaks the file format anyway (#212, #234), is the one chance to fix those places without a second break. Four of the defects were serious enough to file on their own: #304 (two concurrent transactions both commit and one is lost), #305, #306, #307.

The architecture is yours to decide. So instead of one large proposal, this Discussion holds one decision record per topic, each in its own reply below, so that each one can be accepted, changed or rejected on its own thread. Nothing in them is started before you say so, apart from bug fixes for 5.x as today.

Nothing here needs an answer before the questions in #287.

How the records work

• Each record states the context (with the issues behind it), the proposal, the alternatives, what it breaks, what it does to migration, which release it belongs to, and the open questions.
• Status starts as Proposed. You set it to Accepted, Changed (with what) or Rejected, here in the thread.
• An accepted record is copied into the repository as docs/decisions/DR-NNN-<topic>.md, one file per decision, and gets one tracking issue linked from Epic: v6.0.0 — migration guide, changelog, docs, samples and release engineering #296. A later change of mind is a new record that supersedes the old one, so the history of why stays readable.
• Format and public-API changes land only behind an accepted record. Each one ships first in a 6.0.0-preview.N build, and the oracle suites must pass for it.

The one rule behind the scope

The file format may break once. Every change that touches the on-disk format, or the public API, goes into 6.0 together, so that users migrate once. Everything that is internal can keep coming in 6.x minors. That puts each topic in one of three buckets: 6.0 must (format, contract), 6.0 should (internal, but it removes a whole bug class), 6.x (internal improvement).

The decisions

#	Decision	Release	Breaks
DR-1	The oracle suites on main are the gate for every v6 change	before 6.0	nothing
DR-2	Migration from v5: a format guard in 5.3, a read-only v5 reader, a dump format, no in-place upgrade	5.3 + 6.0	nothing for 5.x users
DR-3	5.3 as a bridge release: the new API added next to the old, old members [Obsolete]	5.3	nothing (additive)
DR-4	WAL v2 and transactional storage: every page change logged, one recovery path, real fsync	6.0 must	file format, storage seams
DR-5	Transactions: write conflicts detected, reads from a snapshot	5.x fix + 6.0 should	behaviour (conflict instead of lost write)
DR-6	One index key codec and one sort order for all BSON values	6.0 must	index format
DR-7	One collection object per name, one record store	6.0 must	internal; DynamicCollection constructor
DR-8	One collection core for typed and schema-less collections	6.0 should	internal
DR-9	BSON wire revision and the field-name dictionary	6.0 must	file format, BsonValue API
DR-10	The public API of v6	6.0 must	public API
DR-11	Source generator: diagnostics, incremental models, no run-time reflection	6.0 must / should	generated code (regenerate only)
DR-12	One query plan for LINQ, BLQL and generated filters	6.x	query results where today's paths disagree

Who does what

I'm happy to take DR-1 (the oracle stack is mine), DR-2 (migration) and the documentation side of all of them, and to work on any other one you want help with. Where you'd rather implement a decision yourself, the record and its tests are the hand-over.

[CaffeinatedCoder]

DR-1 — The oracle suites on main are the gate for every v6 change

Status: Proposed · Release: before any 6.0 preview · Breaks: nothing

Context

• The v6 changes below touch roughly 15,000 lines of BLite.Core: storage, transactions, collections, indexes. Example-based tests did not catch the classes behind Typed collections: == null and OrderBy(nullable).Take(n) served from an index return no null-keyed documents (indexes skip null keys, planner reads the null bucket) #247–A failed unique CreateIndexAsync leaves the half-filled index registered #271 or Two concurrent transactions both commit, and one of them is silently lost: no write-write conflict detection #304; the oracle suites did, or would have.
• None of the suites is on main yet: test: oracle tests for the LINQ provider and the typed collection store #250 → test(oracles): model-based test of BTreeIndex with page accounting #259 → test(oracles): crash consistency of the WAL and page store by fault injection #263 → test(oracles): differential and model-based tests of DynamicCollection #272, test(aot): Native AOT smoke test, a consumer under JIT and AOT with the publish log as oracle #277, test: encrypted mode for the store model and WAL crash oracles #294, test: browser harness for BLite.Wasm (OPFS, IndexedDB, in-memory; trimmed and AOT) #295 are open PRs.

Proposal

• Merge the suites and run the fast ones on every PR, the slow ones (AOT smoke test, browser harness, the full crash-state sweep) nightly.
• Add the oracles this review showed are missing:
  • cross-view: the same operations through a typed context, a BLiteEngine and a BLiteSession on one file, checked against one model (The schema-less API cannot read a typed collection that holds a document larger than one page: FindAllAsync and BLiteMigration throw "Skipping type 69" #301, An engine and a session hold separate DynamicCollection instances: an index created through one misses the other's writes, and a session commit can erase the index definition #305);
  • concurrency: two interleaved transactions, expecting both writes or a conflict, never a lost write (Two concurrent transactions both commit, and one of them is silently lost: no write-write conflict detection #304);
  • index vs no index for every differential suite;
  • key codec: memcmp(Encode(a), Encode(b)) equals the comparator, for every pair of edge values (DR-6);
  • round trip: every BSON type and edge value written and read back unchanged (DR-9, BsonValue equality is wrong for negative Int64 and pre-1970 dates: FromInt64(-1) != FromInt64(-1), and FromInt64(0) == FromInt64(long.MinValue) #306);
  • allocation gates for the operations whose allocation the README states.
• Each v6 step is accepted only when the suites pass unchanged, or when a changed expectation is itself a decision in this Discussion.

Alternatives — Refactor first and add tests afterwards: the tests would then describe the new code, not the behaviour users rely on.

Open questions — Which suites in CI on every PR, and with what time budget (#287 asks the same)?

[CaffeinatedCoder]

DR-2 — Migration from v5: a format guard in 5.3, a read-only v5 reader, a dump format, no in-place upgrade

Status: Proposed · Release: 5.3 (guard, export) + 6.0 (reader, import) · Breaks: nothing for 5.x users

Context

• Every v5 database must be converted to v6. Most of BLite's target apps (Where BLite fits, and a reliability-first roadmap #287: offline-first MAUI, Unity, Blazor WASM) hold the user's only copy of the data on the device, so the conversion has to run inside the updated app, unattended, and must never lose data.
• One app cannot load BLite 5 and BLite 6 side by side (same assembly names), so "export with v5, import with v6" alone does not work in-app.
• v5 does not check the file format version. Page 0 carries FormatVersion = 1 (PageHeader.cs:51-54), but no open path reads it; PageFile checks only the page size and the length alignment (PageFile.cs:458-480). A 5.x engine would open a v6 file and write into it: after a downgrade, or when a v5 and a v6 process share a file in multi-process mode.
• BLiteMigration copies through DynamicCollection, which cannot read documents larger than a page (The schema-less API cannot read a typed collection that holds a document larger than one page: FindAllAsync and BLiteMigration throw "Skipping type 69" #301), and leaves the target's sub-files behind when it fails.
• The per-collection required-features record (CollectionFeatures) already refuses features an older version cannot handle, but only from 5.3 on, and older releases overwrite the record.

Proposal

1. 5.3: format guard. Refuse to open a file whose header FormatVersion is greater than 1, or whose WAL header is newer, with an exception that names the version needed. Ship it as early as possible, so that most installed apps have it before 6.0 exists. Same for the browser backends.
2. 5.3: fix The schema-less API cannot read a typed collection that holds a document larger than one page: FindAllAsync and BLiteMigration throw "Skipping type 69" #301 on the read side, so that every v5 tool can read every document.
3. A dump format, documented in docs/spec/: a manifest (collections, id types, index definitions, schemas, TimeSeries and retention settings, KV entries with TTL, field names) plus one stream of standard BSON documents per collection, with full field names. 5.3 can write it (ExportAsync(Stream)), 6.0 reads it (ImportAsync). The same format serves the LiteDB import (Moving from LiteDB: a guide, an API mapping and an import tool #297) and a portable backup. A dump of an encrypted database is itself encrypted with the target's key unless the caller opts out.
4. 6.0: a read-only v5 reader, in a separate package (e.g. BLite.Migration.V5). It reads v5 pages, slots, overflow chains, both C-BSON versions, the dictionary and the catalog, decrypts v5 encrypted files and replays a v5 WAL in memory, without ever writing the source. It feeds the import directly, so an app calls await BLiteMigration.UpgradeFromV5Async(path, options) at start-up. Reading is a small part of the engine: no indexes, no writes, no concurrency.
5. No in-place upgrade. v6 refuses a v5 file and names the upgrade call. The upgrade writes a new database next to the old one, verifies counts and a content hash per collection, and only then swaps the files by rename. The v5 file is kept until the app deletes it. A crash at any point leaves the v5 file intact.
6. Tests: golden v5 files checked in (typed and dynamic, every index type, encrypted, multi-file, with an un-checkpointed WAL, with overflow documents); for each, the v5 engine's reads equal the reader's reads, and the upgraded v6 database equals the store model. Crash injection during the upgrade.

Alternatives

• v6 opens v5 files and upgrades them in place: v6 would carry the whole v5 storage path; a crash mid-upgrade needs its own recovery; no way back.
• Export with v5, import with v6, nothing else: fine for servers and desktop tools, impossible in-app.
• An intermediate app release on 5.3 that exports at start-up: depends on every user installing that release before the v6 one.

Open questions

• One package for the reader, or inside BLite.Core behind a flag?
• How long does 5.x get security and data-loss fixes after 6.0 (Epic: v6.0.0 — migration guide, changelog, docs, samples and release engineering #296 suggests six months)?

[CaffeinatedCoder]

DR-3 — 5.3 as a bridge release

Status: Proposed · Release: 5.3 · Breaks: nothing (additive)

Context — A jump straight from 5.2 to a v6 API means every user rewrites at once, in the same release that changes the file format.

Proposal

• 5.3 adds the v6 API next to the existing one wherever it can be additive: OpenAsync with one options object (DR-10), async replacements for the remaining sync members (DropIndexAsync, RollbackAsync, Configure*Async), CancellationToken overloads, exception types for duplicate key, lock timeout, corruption and transaction conflict that derive from today's .NET bases (so existing catch blocks still work), IAsyncDisposable on the engine.
• The members v6 removes get [Obsolete(DiagnosticId = "BLITE…", UrlFormat = "…")] pointing to the migration guide.
• 5.3 also carries the format guard and the export of DR-2, and the 5.x fixes of The schema-less API cannot read a typed collection that holds a document larger than one page: FindAllAsync and BLiteMigration throw "Skipping type 69" #301, Two concurrent transactions both commit, and one of them is silently lost: no write-write conflict detection #304–BLiteSession.Rollback() does not await the rollback: it returns while the transaction is still active, and failures go unobserved #307.
• A user who moves to the 5.3 API without warnings has only the data migration left for 6.0.

Alternatives — Breaking everything in 6.0 with no bridge: simpler for us, harder for every user, and it mixes code changes with data migration.

Open questions — Is a 5.3 with this much new API acceptable, or should part of it go into a 5.4?

[CaffeinatedCoder]

DR-4 — WAL v2 and transactional storage

Status: Proposed · Release: 6.0 must · Breaks: WAL and header format; IPageStorage / IWriteAheadLog contracts; public StorageEngine page methods

Context

• Two write paths: 38 WritePageImmediate call sites plus PageFile's own header and free-list writes bypass the WAL (allocation, free list, catalog, sequences, schema chain, field-name dictionary, KV store, TimeSeries, index roots). That is the root of B-Tree merge frees the emptied page outside the transaction: rolling back deletes leaves the committed tree pointing at a freed page #233, B-Tree pages allocated by a rolled-back split are never reclaimed #237, PageFile.FreePage is not idempotent: dropping an index while a transaction has merged its nodes frees a page twice #238, A crash leaks pages freed or allocated outside the WAL (queued frees, index roots on first open) #261, A crash between the two page writes of PageFile.FreePage/AllocatePage leaves a Free page off the free list #262 and, in the browser, BLite.Wasm: a committed write is lost on a page reload unless a checkpoint ran, and CheckpointAsync does not reliably prevent it #289.
• The WAL is not fsynced on main (WAL commits are never flushed to disk: a power loss drops acknowledged commits and can leave documents pointing at freed pages #260, fix in fix(wal): sync the WAL to disk before a commit is acknowledged #280); the checkpoint flushes data files without fsync before truncating the WAL; records carry no checksum or sequence number; transaction ids restart at 1 on every open.
• One ReaderWriterLockSlim per file with a 500 ms timeout guards every page write; the checkpoint and backup hold locks for long stretches. That is the source of the lock-timeout flakes (A commit that waits for another commit's disk sync fails with TimeoutException after 500 ms: LockTimeout.Default is tuned for an unsynced WAL #303).
• StorageEngine is 12 partial files, 5,213 lines, ~45 fields: pages, allocation, transactions, cache, group commit, checkpoint, recovery, backup, catalog, schema, dictionary, KV, TimeSeries, sequences, CDC hooks. It depends upward on index types.

Proposal

• WAL v2: records with a log sequence number, a generation salt and a CRC32C; fsync on commit and on truncate; one redo routine for open, multi-process catch-up and the browser.
• Allocation, free list, catalog, dictionary, schema, KV and TimeSeries changes are ordinary page changes inside the transaction. WritePageImmediate and the pending-free machinery are removed.
• One commit path; a checkpointer that owns write-back, fsyncs data files before truncating the WAL and gives CheckpointAsync a real completion; backup as a checkpoint fence plus a snapshot copy, not under the commit lock.
• A narrow IPageTransaction (read, get writable, allocate, free) is the only page API collections and indexes see; storage stops knowing index types.
• One FIFO writer lock plus readers; timeouts only at the public API, through CancellationToken.

What becomes impossible — torn allocations and frees, leaked pages after a crash, a page in neither the WAL nor the free list, losing committed data on a browser reload.

Migration — new WAL and header format; covered by DR-2.

Open questions — Keep multi-process mode in 6.0, or mark it experimental and port it in 6.x?

[CaffeinatedCoder]

DR-5 — Transactions: write conflicts detected, reads from a snapshot

Status: Proposed · Release: conflict detection as a 5.x fix (in progress for #304); snapshot reads 6.0 should · Breaks: behaviour: a conflicting commit fails instead of losing the other one's write

Context

• Two concurrent transactions both commit, and one of them is silently lost: no write-write conflict detection #304: two explicit transactions (or two sessions) that touch the same page both commit, and the first one's writes are lost. Each transaction keeps private page copies, and commit publishes them last-writer-wins. The README promises snapshot isolation; IsolationLevel is stored and never read.
• Reads have no snapshot: a B-tree range scan can skip or repeat entries across a concurrent split; pages are published one at a time.

Proposal

• 5.x: first committer wins. Each transaction records the committed version of every page it writes; a commit whose pages changed since fails with a conflict exception and is rolled back. No format change.
• 6.0: a transaction manager with persisted transaction ids and a versioned page table, so that a transaction reads one consistent snapshot. IsolationLevel is either implemented or removed.

Alternatives — Hold the collection write lock from a transaction's first write to its commit: simpler, but it serialises explicit transactions per collection (per database in single-file mode) and turns long transactions into timeouts.

Open questions — Should a conflict retry automatically for auto-commit writes, or always surface to the caller?

[CaffeinatedCoder]

DR-6 — One index key codec and one sort order for all BSON values

Status: Proposed · Release: 6.0 must · Breaks: index format (indexes are rebuilt by the upgrade); IndexKey semantics

Context

• 9 encoders and 5 comparators decide the order of values in different places. Keys carry no type tag; int, long and double share a prefix with different widths; dynamic doubles are little-endian; strings are not prefix-free (Typed secondary index keys are not prefix-free: values containing bytes 0x00–0x03 are mis-bounded #212); Guids and ObjectIds sort by byte layout, not by value; -0.0 and 0.0 are different keys; NaN sorts differently by key and by CompareTo; BLQL compares Int64 through double. That is the family behind OrderByDescending served by an index returns tied keys in reverse insertion order, unlike LINQ and the non-indexed path #210, Typed secondary index keys are not prefix-free: values containing bytes 0x00–0x03 are mis-bounded #212, DynamicCollection secondary index keys are not unique: deleting entries of a heavily duplicated value is quadratic #234, Typed collections: == null and OrderBy(nullable).Take(n) served from an index return no null-keyed documents (indexes skip null keys, planner reads the null bucket) #247–A NaN with the sign bit clear sorts above +Infinity in an index on a double #257, BLQL filters and sorts cannot reach a nested field, though a dynamic index can be built on one #264–A failed unique CreateIndexAsync leaves the half-filled index registered #271.
• Typed indexes store [key][docId], dynamic indexes store raw keys with duplicates, which keeps duplicate-only code in BTreeIndex (DynamicCollection non-unique index: forward scan misses duplicate keys that straddle a leaf split #213).

Proposal

• One codec, [type tag][payload], order-preserving and prefix-free, with one canonical type order (Min < Null < Bool < Number < String < ObjectId < Date < Guid < Binary < Document < Array < Max); numbers compared exactly across int, long, double and decimal; -0.0 folded to 0; NaN in one fixed place; strings escaped and terminated; descending columns by byte complement.
• Every secondary entry is [k1..kn][docId], so no B-tree holds duplicate keys.
• memcmp is the only comparator. BsonValueComparer, BLQL sorting and the scan path are defined as comparing encoded keys, with a property test (DR-1).
• Null and missing values follow one documented rule, the same on the index and the scan path.

What becomes impossible — indexed and unindexed queries disagreeing because of key encoding.

Open questions — Keep the primary-key encoding as it is, to avoid rebuilding primary trees during the upgrade?

[CaffeinatedCoder]

DR-7 — One collection object per name, one record store

Status: Proposed · Release: 6.0 must · Breaks: internal; the public DynamicCollection constructor

Context

• An engine and a session hold separate DynamicCollection instances: an index created through one misses the other's writes, and a session commit can erase the index definition #305: the engine, every session and the GDPR helper each create their own DynamicCollection for the same name, with their own index list and lock. An index created through one misses the others' writes; a session commit can erase the index definition.
• The schema-less API cannot read a typed collection that holds a document larger than one page: FindAllAsync and BLiteMigration throw "Skipping type 69" #301: only the typed collection handles documents larger than a page; the schema-less one cannot read or write them, so BLiteMigration, Studio and BLQL fail on such databases.
• Slot I/O, overflow, compaction, secure erase and TimeSeries pages are implemented twice, with differences (TimeSeries pages are skipped by the typed raw-BSON paths).

Proposal

• A collection registry per engine: one collection object per name; sessions and helpers get views that carry their own transaction but share the collection's state.
• One record store used by both collection types: slots, overflow chains, compaction and secure erase, TimeSeries pages.
• Index builds that fill the index first and publish it afterwards, under the write lock, failing loudly on unreadable documents.

What becomes impossible — #301 and #305 by construction; a document readable through one API and not the other.

Open questions — none beyond the API shape in DR-10.

[CaffeinatedCoder]

DR-8 — One collection core for typed and schema-less collections

Status: Proposed · Release: 6.0 should (can slip to 6.x) · Breaks: internal

Context — About 2,000 of DynamicCollection's 2,875 lines re-implement what DocumentCollection (4,712 lines) does: delete (pasted four times), retention (copied line for line), vacuum, truncate, CDC (with different id bytes), locking, index maintenance. Of 82 bug-fix commits in src/, 24 touch DocumentCollection.cs and 12 DynamicCollection.cs.

Proposal — One collection core over raw BSON owns CRUD, index maintenance, retention, vacuum, CDC and transactional reads. A typed collection is a generated mapper on top of it; a schema-less collection is BsonDocument on top of it.

Alternatives — Keep both and share only the record store (DR-7): smaller step, the divergence in index maintenance, retention and CDC stays.

Open questions — In 6.0, or as the first 6.x refactor once DR-6 and DR-7 are in?

[CaffeinatedCoder]

DR-9 — BSON wire revision and the field-name dictionary

Status: Proposed · Release: 6.0 must · Breaks: file format (documents are rewritten by the upgrade); BsonValue and dictionary APIs

Context

• BsonValue stores Int64 and dates as double bits, so equality fails for negative values (BsonValue equality is wrong for negative Int64 and pre-1970 dates: FromInt64(-1) != FromInt64(-1), and FromInt64(0) == FromInt64(long.MinValue) #306).
• Decimal is written as decimal.GetBits under the Decimal128 tag 0x13; Guid as a 36-character string; ObjectId in little-endian layout, so _id order is not time order; DateTime with kind Unspecified is treated as local time.
• Field names are lowercased and merged ("Name" and "name" share one id), so schema-less documents lose their case.
• The dictionary is shared mutable state handed to every document and reader, rebuilt O(n) per new name, written outside the WAL, and its ushort id wraps without a check.
• Seven document representations, conversions that decode and re-encode every field.

Proposal

• BsonValue with exact, type-aware equality (5.x fix for BsonValue equality is wrong for negative Int64 and pre-1970 dates: FromInt64(-1) != FromInt64(-1), and FromInt64(0) == FromInt64(long.MinValue) #306; in 6.0 a long field instead of double bits).
• One wire revision: real IEEE Decimal128 (or decimal under a private tag), Guid as binary subtype 4, ObjectId big-endian, an explicit DateTime kind rule, case-preserving field names.
• A dictionary service owned by the engine: immutable snapshots swapped atomically, changes logged in the WAL (DR-4), a hard error at the id limit.
• One immutable raw-document type plus a pooled builder; the unused BsonBufferWriter removed.

Open questions — Real Decimal128, or keep .NET decimal under a BLite-specific tag? Case-preserving names change how schema-less queries match field names: case-sensitive like MongoDB, or case-insensitive as today?

[CaffeinatedCoder]

DR-10 — The public API of v6

Status: Proposed · Release: 6.0 must (additive parts in 5.3, DR-3) · Breaks: public API

Context

• Three entry points with different shapes (BLiteEngine, DocumentDbContext, BLiteSession), twelve constructors, sync and async mixed (13 sync-over-async sites, 34 async methods without CancellationToken), no-op members (BLiteEngine.CommitAsync, Rollback), different dispose semantics (DocumentDbContext.Dispose does not checkpoint, DisposeAsync does; the engine is not IAsyncDisposable), three CDC shapes, four index-info types.
• No exception hierarchy: 152 InvalidOperationException and 71 TimeoutException throw sites.
• 37 public types (409 members) are internals: page structs, StorageEngine, BTreeIndex, WalSharedMemory.

Proposal

• One database type opened with OpenAsync(options); typed and schema-less collections as two views of the same collection (DR-7); one shared collection contract; database-level operations (backup, vacuum, checkpoint, list, drop) in one place; DocumentDbContext stays as the generator's target and exposes the database.
• Async-only I/O with CancellationToken last; no no-op or fake-async members; IAsyncDisposable with the same close semantics everywhere.
• One options object (storage, locking, encryption, GDPR, KV, metrics, audit, multi-process); CryptoOptions stops exposing the passphrase.
• A BLiteException hierarchy with error codes (duplicate key, validation, lock timeout, corruption, transaction conflict, not supported).
• Internals made internal; generated-code support types in their own namespace; [Experimental] on multi-process mode and the browser backends; package validation with a 6.0.0 baseline.

Open questions — Keep the name BLiteEngine or introduce a new one? Which of today's members are important enough to keep as [Obsolete] shims through 6.x?

[CaffeinatedCoder]

DR-11 — Source generator: diagnostics, incremental models, no run-time reflection

Status: Proposed · Release: diagnostics and key tables 6.0 must; rewrite 6.0 should · Breaks: generated code (regenerate only)

Context

• The generator silently skips contexts not named *Context (A DocumentDbContext whose class name does not end in "Context" gets no generated code and no diagnostic; its collections stay null #300) and reports skips as #warning text inside generated code; all diagnostics are at Location.None.
• Incremental caching never hits (models without equality, the whole compilation collected), and code is emitted by string concatenation in about 2,900 lines.
• Generated code can fail to compile for property names such as Entity, Reader or Default, and for more than 255 properties; types are classified by name suffix.
• The typed path still reaches run-time reflection through BsonSchemaGenerator.FromType<T>() under suppressed IL warnings, which is how Renamed properties ([Column]/[BsonProperty]/[JsonPropertyName]) fail on insert: field name not registered #184 happened.

Proposal

• A diagnostic with a source location for every skip and every unsupported shape.
• Generated key tables and schemas, so the typed path needs no reflection at run time.
• Equatable models, a model → emitter split, snapshot tests of the generated code.

Open questions — Keep the netstandard2.1 reflection fallbacks, or require the generator on every target?

[CaffeinatedCoder]

DR-12 — One query plan for LINQ, BLQL and generated filters

Status: Proposed · Release: 6.x · Breaks: query results where today's paths disagree (each such change goes through the differential suites)

Context — Four query front ends (LINQ, BLQL, generated {Entity}Filter, count and aggregate fast paths) each choose indexes their own way; there is no logical plan; an exact index result skips the residual predicate, so index-key semantics replace LINQ semantics; Count() and ToList().Count follow different rules.

Proposal — LINQ, BLQL and generated filters compile to one predicate representation plus sort, skip, take, projection and aggregate; one planner picks the access path; one executor evaluates the residual over raw BSON; an index result is treated as exact only when the planner proves it. Reference semantics are defined once: LINQ-to-Objects for typed queries, the DR-6 order and null rule for BLQL.

Alternatives — Keep the front ends and fix divergences one by one, as now: each fix is small, the class does not close.

Open questions — Does BLQL keep MongoDB-style semantics where they differ from LINQ (missing fields, array matching)?

[CaffeinatedCoder]

Withdrawn by the author on 2026-10-19; the reasoning is in the comments on #288 and #296.