v0.6.1

ChalkQL v0.6.1

A patch release for long-running hosts with standing and incremental views. An execution arena now keeps what its executions need to run again without the heap, and no more, so a ceiling is all a host sets. Managed output is copied out of the arena rather than taken from it, and a standing view stops holding memory for rows it no longer has.

Targets .NET 10. A JDK 21 or newer is needed only to run the planner locally. Assemblies and namespaces retain the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Elastic License 2.0	engine, planner client, catalogue, execution, entitlements and standing views
ChalkQL.Sources	Elastic License 2.0	POCO, Akade.IndexedSet, ADO.NET, DuckDB and source conformance implementations

Arenas keep their working set

An execution rents arrays of several sizes at different times. To run again without the heap, it needs as many arrays of each size as it ever had rented at once, and that can exceed its peak. A statement that ran once a minute with RetainBytes above its peak still fell back to the heap by about 49 MB a run, because each trim let go of arrays the next run rented again.

• ExecutionArena.WorkingSetBytes reports that need: for each size class, the most arrays rented at once, summed, and the largest over the executions the arena has served.
• An arena keeps its working set, up to RetainBytes. RetainBytes becomes a ceiling on memory rather than a measure of need. With only a ceiling set, a repeated statement rents nothing from the heap once warm, and an arena whose executions need little keeps little.
• ArenaOptions.WorkingSetMargin, 1% by default, is room the trim leaves before letting go. It is not memory kept: the pools hold only what executions gave back, so they settle at the working set.
• A result given back late no longer pushes the pools over their budget. The trim counts what is still outstanding.

The engine's worst case is unchanged, at ArenaPoolSize × RetainBytes.

Managed output is copied out

Under OutputMemory.Managed, the default, a projection's, filter's or join's batches were built in arena scratch, and the arrays were then handed to the host. They never came back, so every run allocated its output afresh, at a power of two. Batches are now copied out at their exact size into arrays of the host's, and the scratch stays pooled for the next. A filtered scan and a join that fell back 9.6 and 19.3 MB over three repeats now fall back nothing. POCO tables' string and validity buffers are copied the same way.

Standing views give memory back

A maintained view's stores kept the room their fullest moment needed. Removals freed slots for reuse but never shrank the arrays, so a view that lived through a compaction held what its inputs held at their largest. In one host, a long-lived view grew by about 15 MB an hour while a fresh view held steady.

• A view whose state has mostly emptied builds it afresh: at its next update, or beside the running state under LowJitter. Readers are sent only the keys that differ. A state is sparse once it holds a quarter of its slots or fewer, over at least 65,536, so a rebuild follows a large emptying, not every compaction. A rebuild that fails is not retried.
• State grows by half again rather than doubling, and a key map that compacts gives back its spare buffers and table.

Through hours of appends and compactions, a latest-row view holds 1.3 times what a fresh view over the same rows holds, where it held 1.75 times before, or many times more after a compaction.

Changed

• Arenas retain less by default. An arena whose working set is below RetainBytes (64 MiB unless set) now keeps its working set, not the full RetainBytes. Set RetainBytes to the most you want an arena to keep.
• A standing view may rebuild itself after its state empties, sending readers the keys that differ, as it does when it moves to a new plan.

Compatibility

• Additive API only: ExecutionArena.WorkingSetBytes and ArenaOptions.WorkingSetMargin. A negative margin is refused.
• No change to the protocol, the IR or the planner. A 0.6 planner serves a 0.6.1 client.

Full Changelog: v0.6...v0.6.1

Licence and attribution

ChalkQL is source available under the Elastic License 2.0 (ELv2).

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.