v0.6

ChalkQL v0.6

This release lets standing views and refreshes run side by side. A view's update that a refresh overtakes is published at the version it read rather than thrown away, and no refresh waits for a view. A refresh that writes nothing a view reads moves the view on without computing. Every commit a reader receives carries a stamp, so a host can wait until its own change shows. A refresh can be marked a barrier that every view publishes itself, and a view can publish every version in order.

Parameters of a view can now be bound again without planning the statement anew. Incremental maintenance, still experimental, gains window functions, index lookups over a range, and filters over the clock that let go of the rows they can no longer pass. The planner and executor fix a handful of statements that were previously refused; Changed lists the results that move.

Targets .NET 10. A JDK 21 or newer is needed only to run the planner locally. Assemblies and namespaces retain the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Elastic License 2.0	engine, planner client, catalogue, execution, entitlements and standing views
ChalkQL.Sources	Elastic License 2.0	POCO, Akade.IndexedSet, ADO.NET, DuckDB and source conformance implementations

Highlights

Views and refreshes, side by side

• An update is published at the refresh it read.
  • An update pins its tables when it begins, and the pin names the refresh it read.
  • A refresh that commits while the update runs no longer discards it. In 0.5, after two attempts, the update held every refresh off until it finished.
  • The view publishes what it computed at that refresh's version, then catches up at once, three times at most, so a quiet feed still converges.
  • Every commit is a state the engine really held, and versions only increase.
  • With refreshes every 10 ms against a 50 ms update, the longest refresh fell from 103 ms to 17–30 ms; with no view it is 8 ms. In a pivot sample with slow views updating beside it, the pivot's 99.5th-percentile latency fell from 448 ms under the old retry to 26–28 ms, level with no views at all.
• A refresh that writes nothing a view reads relabels it. None of the view's tables, and not the clock where the view reads it: the view moves to the refresh's version without computing, and readers get a commit at it with the same checksum. In a practice sample, untouched views' updates fell from 0.19–0.30 ms to almost nothing at the median, and its fan-out roughly halved.
• Stamps and waiting.
  • A sink may implement Commit(in ViewStamp stamp, uint checksum). The stamp is the view's version and its generation, which counts its rebinds. Commit(long, uint) remains the default, so existing sinks compile unchanged.
  • view.Stamp is where the view stands.
  • await view.UpdateToAsync(stamp) returns once the view stands at or after the stamp, waiting for a version the engine has not published yet.
  • ChalkEngine.VersionOf(epoch) gives the version of a recent epoch, such as the one RefreshAsync returns.
• Barriers. RefreshBuilder.Barrier() marks a refresh, such as a day-end close, that every view publishes itself rather than going past. An update that reaches one stops there; the next carries on.
• Every version, opt-in. ViewOptions.Versions = ViewVersions.Every makes a view publish every refresh that changes what it reads, in order, one per update. A row added in one refresh and removed in the next reaches a reader as both changes, the same on every run. The engine holds each refresh's state of the view's tables until the view has published it, at most MaxBacklog refreshes (1,024 by default); one more faults the view by name.
• A reader still drains at its own pace. A view keeps the latest row of each key, so a reader sees each version of a barrier or an Every view only if it drains between updates. The guide shows a pipe of reusable batches for a slow reader.

await engine.RefreshAsync(r => r.Version(900).Barrier().Replace(closes, closing));

while (view.Version < engine.Version)
{
    await view.UpdateAsync();
    view.Drain(ref cursor, sink);   // every reader, before the next update
}

Rebindable parameters

A view names the parameters it may bind again, and RebindAsync moves them. The view keeps its plan, and the planner is not asked for anything:

var view = await engine.MaterializeAsync(sql, new ViewOptions<PointKey, PointRow>
{
    Key = ["bucket"],
    Parameters = new { level = 5_500.0, picks = new[] { "EUR", "GBP" } },
    Rebindable = [new("level"), new("picks", Capacity: 32)],
    // …
});

ViewStamp stamp = await view.RebindAsync(new { level = 5_520.0 });

• Maintained incrementally, only the operators that read a moved value run again, over what they hold, and readers get the keys whose rows changed. Compared with a time, date or integer column, as in WHERE ts >= @from, a move reads again only the rows between the old value and the new.
• A list after IN keeps the length it was bound with. Given a capacity it binds any list from one element up to it into the same plan.
• A WITH of a parameter, such as WITH t AS (SELECT CAST(@level AS DOUBLE) AS level), is the parameter once planned, so it is rebound the same way.
• Refused by name.
  • When the view is made: a name the statement does not have, the clock's, or a parameter a maintained LIMIT or as-of seek fixes at the start.
  • At a rebind, before anything moves: a name the view did not declare, or a list it cannot take. A rebind whose update fails leaves the view as it was.

Incremental maintenance, experimental

Incremental maintenance still carries the diagnostic CHALK003. It now also maintains:

• Window functions whose frame ends a bounded distance from the current row: LAG, LEAD, ROW_NUMBER, RANK, DENSE_RANK, FIRST_VALUE, and running aggregates.
  • An update computes again only the rows a change can reach, and reads only the rows those read.
  • A running call resumes from the value it held at the row before.
  • A running sum over 128,000 rows in eight partitions took 0.2 ms an update for a row appended to each, against 204 ms to recompute. A LAG over a ten-minute window reads at most 30 rows an update.
• Index lookups over a range, kept as a filter over the table.
• Filters over the clock that let go. Where a cut-off such as ts > CURRENT_TIMESTAMP - INTERVAL '2' HOUR bounds the whole condition from below, the filter lets go of the rows behind it. A two-hour window over a day of marks holds about a sixth of the log.

Changed

Views

• More commits. A view now sends readers a commit at a version it relabels without an update, with the same checksum. A reader that treated every commit as a change of rows sees more of them.
• No hold on refreshes. An in-process view's update no longer holds refreshes off. A view whose update pinned no table, such as one that reads only a database, keeps the old retry.
• Views over databases are published at the version of the in-process tables they read. Their database rows are as of when they were read, which may be later. Such a view goes past a barrier, and Versions = Every is refused for it.

Results

• Timestamps cut to their precision. A cast to TIMESTAMP(0)–TIMESTAMP(3) at run time, and a value bound to such a parameter, kept digits beyond the declared precision. CAST(@now AS TIMESTAMP) gave 00:05:00.123 for 00:05:00.1234567. Both executors and the binder now cut to the declared digits, towards the past, as a literal the planner folds already did.
• Set operations over different precisions cast each input to the set's type, so a UNION ALL of a TIMESTAMP(9) and a TIMESTAMP(0) plans where 0.5 refused it. A composite or list column is carried as it is.

Errors

• An untyped parameter is refused with a message that says how to type it, keeping the planner's own words.
• A reserved word used as a name is named in its refusal.

Fixed

• A parameter in a CASE branch, as in CASE WHEN ? < 0 THEN x / ? END, takes the type its call infers, where 0.5 refused it.
• A window's constant argument may be a parameter, as in LAG(qty, 1, CAST(@fallback AS BIGINT)), where 0.5 raised an internal error.
• Hash joins with many rows per key build in linear time. An as-of join over four keys of 40,000 rows took 4.4 s; it takes about 50 ms.
• An index backed by a POCO table's own order extends by the rows an append adds. On a log of 1.6 million rows, an append of 8 rows takes 0.07 ms where it took 11.5 ms.
• Planner catalogs are isolated by host. Engines that share a planner and a ContextId no longer share its catalog entry.
• The planner sub-process is reaped when its host terminates, however the host ends.

Known limits

• Incremental maintenance is experimental under CHALK003, and may change in any release.
• Windows, maintained:
  • Refused, because one change moves every row of the partition: a frame that reaches the end of its partition, an aggregate OVER (PARTITION BY …) with no ORDER BY, NTILE, PERCENT_RANK and CUME_DIST.
  • A call a tie decides (a ROWS frame, LAG, LEAD, ROW_NUMBER, FIRST_VALUE) is refused unless its partition and order keys cover a unique key the input declares.
  • A RANGE frame with an offset reads from the partition's start.
  • A call that reads the clock or a rebindable parameter computes every partition again when it moves.
  • A LAG or LEAD default that is a column, not a constant, is refused as an internal error.
• Rebindable parameters:
  • A filter on a table that reads one holds every row of the table, about 300 bytes a row for four narrow columns, and a move runs over all of them.
  • An empty list is refused under a capacity.
• Every version and barriers reach a reader only if it drains between updates.
• As-of joins, maintained: the right side must be a table read through filters and projections.
• SQL:
  • A named parameter used where it takes two different types cannot be bound. Write a literal of the wider type beside it, as in @trigger < 0e0.
  • There is no MAX_BY.
  • BOOL_OR and BOOL_AND are not yet offered to SQL.
• Unchanged from 0.5:
  • ChalkQL reports the first rule a statement breaks.
  • No index serves ILIKE.
  • Composite types are one level deep.
  • CHALK001 still marks the experimental arena APIs.
  • ChalkQL remains read-only, and some correlated and LATERAL query shapes are refused.

On the wire

• No change to the protocol or the IR. The IR version does not move, so a 0.5 planner serves a 0.6 client. It lacks the planner's fixes above, though: deploy the 0.6 planner JAR to a remote planner. PlannerArtifact.CopyToAsync exports the artefact matching the installed client.

Full Changelog: v0.5...v0.6

Licence and attribution

ChalkQL is source available under the Elastic License 2.0 (ELv2).

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.