35.12.1

Added

• LsmKeyValueDBOptions.MaxTransactionMemory (256 MiB by default) bounds the memtable bytes one LsmKeyValueDB
  writing transaction adds. A transaction above it throws BTDBException on its next modification or commit and can
  only be rolled back, so memtable memory stays bounded.

• The low-level interpreter has frame opcodes (OpCode 152–162): reading key or value fields straight into frame or
  object slots, repeated field skips, typed compare-and-jump against constants (QueryValueType, QueryCompare),
  string compare-and-jump, and slot conversion (QuerySlotKind). They use fixed-width operands; InterpreterCtx gains
  Frame, Objects, KeyReader and ValueReader refs for them.

• IRelation.Query<T>() Where predicates now support:

  • comparisons, !, bool members and comparisons between properties;
  • arithmetic, numeric and enum casts;
  • nullable values with lifted C# semantics;
  • Guid, decimal, TimeSpan, DateTimeOffset and enum fields;
  • string StartsWith, EndsWith, Contains, Equals, Length, string.IsNullOrEmpty and
    string.CompareOrdinal;
  • Contains on constant collections.

  String values are tested without allocating. Predicates keep C# short-circuit exception semantics. See
  Doc/Relations.md.

• IRelation.Query<T>() supports terminal operators:

  • Count and LongCount, which never materialize rows and use key indexes in O(log n) without a residual
    predicate;
  • Any and All;
  • First, FirstOrDefault, Single, SingleOrDefault, Last and LastOrDefault, which materialize only the
    returned row;
  • Min, Max, Sum and Average with a selector, which read only the needed fields.

  Results and exceptions match LINQ-to-Objects.

• IRelation.Query<T>() plans access paths. These become key range seeks instead of row tests:

  • equalities and Contains / || lists on leading key fields, as several seeks merged in key order;
  • <, <=, >, >= on integer, bool and enum key fields;
  • ordinal StartsWith on string key fields.

  Secondary indexes are chosen when they allow a longer prefix or range than the primary key, still returning rows in
  primary key order. Count, Any, All, Min and Max read only a covering index.

• IRelation.Query<T>() supports OrderBy, OrderByDescending, ThenBy, ThenByDescending, Skip and Take
  after Where, with LINQ-to-Objects results and exceptions:

  • orderings by key fields stream in key order;
  • other orderings sort keys read from stored bytes, keeping only Skip + Take rows;
  • unfiltered Skip uses key indexes;
  • only returned rows are materialized.

• IRelation.Query<T>() supports Select after Where and ordering, followed by Skip, Take, enumeration or
  terminal operators:

  • single values, anonymous types, constructors, ValueTuple / Tuple, records and member initializers;
    [Generate] classes are filled through generated setters or by writing auto-property backing fields in place,
    and value arguments are not boxed per row;
  • only the selected fields are read, and only for returned rows;
  • Min, Max, Sum and Average without a selector aggregate projected values.

  Query<T>() now returns IOrderedQueryable<T>. Selector calls and exceptions match LINQ-to-Objects.

• BTDB.SourceGenerator intercepts IRelation.Query<T>() query chains written inline (Where, Select, ordering,
  Skip, Take and terminal operators) when a project adds BTDB.Interceptors to InterceptorsNamespaces, with a
  .NET 9 SDK compiler or newer. Intercepted operators skip building their expressions, and repeated executions read
  captured values directly instead of walking the expression: a point query takes 0.96 µs instead of 2.1 µs. See
  Doc/SourceGenerator.md.

• The low-level interpreter has expression opcodes (OpCode 163–185):

  • frame-to-frame compare, arithmetic (QueryArithmetic, QueryOperandKind) and numeric conversion;
  • capturing raw field bytes, with encoded equality and StartsWith;
  • string decoding into reused buffers and string operations (QueryStringEncoding, QueryStringOp,
    QueryStringSet);
  • set membership, query exceptions (QueryException) and skipping fields by field handler;
  • copying selector results (CopyToFrame, 183);
  • decoding captured strings into object slots (StringFromCaptured, 184) and clearing object slots
    (SetObjectNull, 185).

  QueryValueType gains Guid, Decimal, TimeSpan and DateTimeOffset.

Changed

• ILsmKeyValueDB.WaitForDurabilityAsync and LsmKeyValueDB.WaitForDurabilityAsync accept an optional transaction
  number. Omitting it or passing null waits for the latest transaction committed at the time of the call, without
  including later commits or uncommitted changes.

• IRelation.Query<T>() predicates evaluate 2–4× faster per scanned row:

  • one interpreter program per stored value version;
  • fields are read only up to the last one a conjunct needs, and each conjunct is tested right after its last field
    is read;
  • equalities already guaranteed by the primary-key cursor prefix are not re-checked;
  • rows are tested with FastIterate;
  • row loaders are cached per type.

  Selective queries are now faster than enumerating the relation and filtering with LINQ.

• The interpreter runs programs without try blocks in an instruction loop without exception bookkeeping, and no longer
  enters a try block per instruction for programs that have them.

• IRelation.Query<T>() throws NotSupportedException for Where with an element index.

• IRelation.Query<T>() caches compiled queries per relation by expression shape, with captured values and
  Skip/Take counts as parameters bound for each execution. The cache is bounded to 256 shapes per relation.
  Repeated queries skip expression conversion and compilation: a point query takes 2.1 µs instead of 3.3 µs and
  allocates 3.7 KB instead of 10 KB, and with Select 4.1 µs instead of 7.0 µs. Scanners rent their value buffer
  from ArrayPool, and captured arithmetic such as id + 1 is evaluated without the expression interpreter.

• IRelation.Query<T>() scans are 5–15 % faster:

  • selected properties are read straight into their result slots;
  • common integer fields are decoded, and common fields skipped, without the generic RunCodeOp dispatch;
  • whole-prefix scans no longer re-check their keys.

Fixed

• Fix LsmKeyValueDB.CompactorReadBytesPerSecondLimit being ignored; it now paces the reads of compaction inputs.
• LsmKeyValueDB compactor read and write limits counted time since the database opened (or up to 10 seconds of
  idle time) as unused budget, so a compaction after idle time ran unpaced until it used up that credit.
• DiskChunkCache.Dispose threw NullReferenceException when a hash index file it knew about had already been removed
  from its file collection.
• Source generator:
  • incremental generation compared only kind and name of a type, so edits (new properties, relation methods, moved
    errors) kept stale generated code in IDEs or crashed the generator; EquatableArray.Equals(object) overflowed
    the stack;
  • decimal primary keys and UpdateById values, and IList<string> / IList<ulong> key parameters threw
    NotSupportedException at runtime;
  • FindBy returning ICollection<T> or IReadOnlyCollection<T> threw InvalidCastException;
  • relation methods re-declared with new (e.g. RemoveAll, UpsertRange) threw NotImplementedException;
  • a read-only property whose getter returns a backing field was treated as stored, so the relation failed to open;
  • IOrderedDictionaryEnumerator type arguments lost global:: and failed to compile in namespaces like
    Collections.Model;
  • non-public [OnSerialize] / [OnBeforeRemove] methods in generic classes generated code that did not compile.
• Relations:
  • RemoveById(prefix, AdvancedEnumeratorParam) in descending order removed rows outside the requested range;
  • CountBy / AnyBy with an excluded start skipped only one key when several rows shared the start prefix;
  • Constraint<byte> / Constraint<sbyte> (Any, generic Exact, also nullable) used variable length encoding
    although these keys are stored in one byte, so values ≥ 128 or negative were never found;
  • IOrderedDictionaryEnumerator.NextKey restarted from the beginning after reaching the end, and Count /
    Position reset to 0 there (setting Position back threw NullReferenceException);
  • loading a value type bigger than 16 bytes through a property with a custom getter or setter threw
    NullReferenceException;
  • IL emit relations (UseNoEmitForRelations = false): FirstById / LastById with constraints threw
    InvalidProgramException and [OnBeforeRemove] with container resolved parameters threw ArgumentException.
• ObjectDB collections and field handlers:
  • IOrderedSet and IRoaringBitmap enumerators restarted from the first item when MoveNext was called again
    after the end;
  • IOrderedSet with a tuple key such as (int, byte[]) failed on reopening the database;
  • IL emit: upgrading byte[] items to ReadOnlyMemory<byte> / ByteBuffer and IOrderedDictionary with
    ReadOnlyMemory<byte> keys or values threw.
• Key value DB:
  • InMemoryKeyValueDB cursors stayed valid after a failed FindFirstKey, FindLastKey or Find, FindKeyIndex
    with a prefix could return a key outside of the prefix, an invalid cursor did not start from the last key on
    FindPreviousKey after another cursor wrote, and FastIterate threw after its last key was erased;
  • BTreeKeyValueDB cursors restarted FastIterate from the first key, threw from SetValue and returned a wrong
    KeyHasPrefix after another cursor of the same transaction wrote;
  • BTreeKeyValueDB.IsCorruptedValue reported values written in the current transaction as corrupted;
  • KeyValueDBExportImporter.Import / ImportAsync failed on streams returning fewer bytes than requested (e.g.
    GZipStream);
  • one DiskChunkCache file with a truncated header made the whole cache unloadable.
• Event store:
  • a cyclic object graph overflowed the stack in TypeSerializers and event store v1;
  • struct properties bigger than 16 bytes (e.g. decimal?, Guid?) with a custom getter or setter threw
    NullReferenceException;
  • event store v2 could not store System.Tuple<>, tuples with enum items, or re-serialize dynamically loaded
    objects, and in IL emit mode corrupted memory for value tuples with class items;
  • reading tuples with unknown or unregistered item types and decimal? / Guid? / tuple nullables as dynamic
    objects threw; setting a member of a dynamic object or tuple threw InvalidCastException.
• Streams, collections and compression:
  • ReadOnlySequenceMemReader read values split across segments wrongly and ReadOnlyMemoryMemReader did not
    advance after ReadByteArrayAsMemory;
  • BrotliDecompressMemReader reported wrong positions, overwrote reads bigger than 64 KiB and threw at the end of
    a source without trailing data; BrotliCompressMemWriter reported wrong positions after 64 KiB;
  • MemWriter.WriteStringInUtf8 threw when a multi-byte character crossed a buffer boundary of a stream backed
    writer; on corrupted data ReadString / SkipString could read past the string instead of throwing
    InvalidDataException; CheckMagic consumed data on end of stream;
  • LruCache and SpanByteLruCache lost entries after IncreaseCapacity or Clear following a Remove, and
    SpanByteLruCache lost keys after compaction;
  • EquatableArray.Equals(object) overflowed the stack, StructList.AddUnique(null) threw, an empty
    Utf16Utf8Dictionary threw on lookup and ByteStructs.Key32 read its last field from a wrong offset;
  • FastBehaviourSubject.Subscribe could deliver the current value twice when the observer re-entered the subject;
  • Snappy decompression of literals with a 4-byte length and compression of empty input were broken.
• IRelation.Query<T>() ignored the StringComparison of string.Contains(char, StringComparison), so for example
  OrdinalIgnoreCase did not match a differently cased character.
• IRelation.Query<T>() threw InvalidCastException for Contains on a collection of bool.
• IRelation.Query<T>() equality conditions on decimal, DateTime and DateTimeOffset primary or secondary key
  fields searched for the exact stored key bytes, so values equal in C# but differing in decimal scale, DateTime.Kind
  or offset (for example 0.250m and 0.25m) were not found; such conditions are now tested on every row.
• RawData.GetSizeAndAlign guessed the alignment of structs from their size, so DateTimeOffset? fields of relations,
  objects and events (and nullables, value tuples and key value pairs of other structs whose size is not their
  alignment) were saved and loaded at a wrong offset: they could not be read back, typically failing with "Offset must
  be within plus or minus 14 hours". Values of such fields saved by earlier versions are not readable.
• On x86/x64, MemReader.ReadStringOrdered, ReadStringOrderedAsSpan and PackUnpack.ReadAndExpandSimpleCharacters
  wrote past the end of a character buffer shorter than 16 characters and lost characters (read as \0) when it grew;
  their vector stores also wrote through an unpinned pointer into heap buffers.
• IL: the caching IL builder treated 0.0 and -0.0 constants as equal, and Newobj with a non-public
  constructor in a dynamic type threw MethodAccessException.