Skip to content

v0.2

Choose a tag to compare

View all tags
@markhammond markhammond released this 23 Sep 10:47
· 55 commits to main since this release
v0.2
bb88675

ChalkQL v0.2

The second public release. Three themes: the planner now plans for the rows a request will
actually pull, a source can be an Akade.IndexedSet with every index it carries, and a bound or
a value that is a parameter no longer costs the plan anything a literal would not.

Targets .NET 10. A JDK 21 or newer is needed only to the planner locally. Assemblies and namespaces keep the Chalk.* names.

NuGet packages:

Package Licence
ChalkQL NuGet Apache-2.0 engine, planner client, catalogue, execution and entitlements
ChalkQL.Sources NuGet Apache-2.0 POCO, ADO.NET, DuckDB and source conformance implementations

Highlights

An Akade.IndexedSet is a table

Chalk.Sources.Akade, shipped in ChalkQL.Sources, publishes an
Akade.IndexedSet as an ordinary SQL table and
discovers the indexes it can serve faithfully, with no adapter written by the host:

var source = AkadeSource.From("purchases", purchases).Build();
  • Hash and ordered indexes on one member or on a tuple of two to four, written either as a
    lambda tuple or as a named key method declared with CompoundIndex. A hash index reports its
    exact distinct key count to the planner.
  • Key types and comparers. Integer, decimal and temporal keys are ordered access paths on the
    strength of the type; float, double, string and Utf8String become ordered once the
    comparer the index was built with is declared, ChalkComparers.For<T>() or
    StringComparer.Ordinal. A descending comparer registers a descending index, which serves
    ORDER BY … DESC without a sort.
  • LIKE 'p%' is an index lookup. An ordered string index is sent the half-open range the
    prefix stands for; Akade's trie is a new PREFIX index kind and is sent the prefix itself.
  • An ordered index is read backwards for a range with no upper bound, so
    ORDER BY ts DESC LIMIT 1 is one walk and one row rather than a sort of the table.
  • Ordered lookups stream lazily as Akade produces them, under a per-row order guard that
    refuses an out-of-order row by name; a consumer that stops after one row pays for one row.
  • Lifecycle. The one rule a host keeps, no mutation overlapping an execution or a refresh,
    and the three ways to keep it: in place between requests, swapping the set behind a delegate,
    or the transactional Append and Replace while requests are in flight. The package README
    opens with a table of what each Akade index becomes, and a benchmark measures ChalkQL's
    execution overhead over an Akade set against Akade's own calls.

Plans that fit the request

  • A limit reaches the leaf that can stop early. A LIMIT above a scan or an index lookup,
    through a projection and a filter, gives the leaf a row goal: the planner costs the leaf for
    the rows the limit will pull rather than its whole output, so a statement that wants one row
    out of an ordered index seeks once. The goal travels to sources as ScanRequest.RowGoal and
    IndexLookupRequest.RowGoal, a hint never a bound: the fetch above stays authoritative. The
    in-memory source sizes its first batch by it and doubles from there.
  • A bound above a UNION ALL or a partitioned table is copied into every branch, and a
    branch that is one remote source's subtree carries the copy away as that source's own
    ORDER BY … LIMIT.
  • LIMIT ? and OFFSET ? plan and execute. The bound travels as the parameter it is and is
    read when the execution starts; a negative value, a NULL and a fraction are each refused by
    name before any row moves.
  • A prepare may say what it expects its parameters to be worth.
    PrepareAsync(sql, parameterValueHints, …) takes the same container an execution binds from,
    sparse, with null meaning nothing said and DBNull.Value meaning SQL NULL. The planner
    estimates a predicate against a hinted parameter from the column's statistics instead of
    guessing, and a hinted LIMIT ? gives the leaf its row goal. A hint informs an estimate and
    never a truth: the plan's tree, its parameters and its rows are what they would have been, the
    values bound at execution need not match, and a hint's value reaches no log line, exception,
    plan text or digest. One saved filter, WHERE amount BETWEEN ? AND ? ORDER BY unit_price LIMIT ?,
    plans ordering-first for an interactive grid and predicate-first for a report export, from the
    hints alone.
  • A parameterised bound is pushed to a source where a literal one is. The generated SQL
    carries a placeholder where the dialect puts the count — LIMIT ?, FETCH NEXT ? ROWS ONLY,
    TOP (?) — and the executor writes the value bound at execution into the text before the query
    is sent, one mechanism for every dialect. The guide's federation section shows how a host
    enables limit pushdown and what each source is then sent.

Entitlements

  • A grant confined along another tenancy kind applies to a table that holds one kind on its own
    row and inherits the other along a declared path    : a reviewer for one supplier, only in one
    warehouse, over positions that carry the warehouse and reach the supplier through their
    product. The confinement is decided above the join, over the two rows together; column rules,
    both binding modes, the explain and the reconciler follow. Such a grant was refused before.
  • A refusal names its near miss. A confinement that would have to be carried along a
    Related path is refused with the model that comes closest and why it is not close enough.

Safe to log

  • A redacted marker can say where its value came from. A literal that is, in type and value,
    one of the request context's bound values prints the name it was bound under beside its
    pseudonym, /*REDACTED-3f9a1c2e:DECIMAL @ctx.tenant_id*/, in the plan text and in the query
    text a source failure quotes. A parameter written as @name reads as @name again in the
    redacted statement and plan text rather than as the positional placeholder the planner saw.
    The structural hash and every pseudonym are unchanged.

Faster

  • GROUP BY on two or more keys is two to three times faster: a scan under a computing
    projection reads only the columns the expressions name, the field trimmer no longer keeps a
    declared collation's columns alive, and compound aggregate keys take the fixed-width image path.
  • A top-N's heap pass is priced, so a top-N over a hundred thousand rows is no longer costed
    like one over ten.

Fixed

  • A LIMIT no longer claims its input's ordering upward, which could let a sort above it be
    removed although it decided the answer.
  • A redacted plan text over a SEARCH — any IN list of two or more — no longer fails with an
    internal error.
  • SELECT … LIMIT ? no longer fails inside the planner with an assertion where support belongs.
  • A source whose dialect cannot write an OFFSET is no longer handed one; Calcite's SQL Server
    dialect spells a bound TOP (n) and dropped the offset beside it, returning the wrong rows.
  • A UNION ALL under a parameterised bound with an OFFSET no longer fails planning.
  • A hash index whose key a scan does not project in full is no longer offered as a lookup over a
    key prefix.
  • A range's bounds are stated in the index's own key order, so a descending key column no longer
    has its > and < bounds the wrong way round.
  • The Akade adapters call only the operations that honour the comparer an index was built with;
    Akade's one-sided range calls do not, and answered an index whose order is not the CLR default
    with no rows.

Changed

  • An Akade source's table is named after the source unless TableName says otherwise. The
    previous default was the word data. A host that relied on that name must now name it.
  • The Akade sample shows only what a host does: AkadeSource.From over the README's example
    sets. Its hand-written index adapters, which the package superseded, moved into the test kit.
  • A remote query's parameters are listed in textual order, one per placeholder, which is what
    a bound at the front of a statement needs. No recorded plan moved.
  • The public API of Chalk.Sources.Akade is tracked in PublicAPI.Shipped.txt and
    PublicAPI.Unshipped.txt like every other assembly.

On the wire

Every addition is additive and a plan that carries none of them is byte-identical to its 0.1
form: Read.row_goal, IndexLookup.row_goal, IndexLookup.reverse, Fetch.count_param,
Fetch.offset_param, TopN.count_param, TopN.offset_param, RemoteQuery.rendered_bounds,
IndexRange.prefix, IndexKind.PREFIX, Index.reversal, InheritedVisibility.path_predicate,
ExplainedPath.path_predicate, PlanRequest.parameter_hints, RedactSqlRequest.context. The
planner's configuration hash moves with the cost model, so a host cannot mistake two sidecars that
plan differently for two that plan alike.

Known limits

  • A bound with an OFFSET beside it is not copied into UNION ALL branches; the bound stays
    local above the union.
  • SUM over an integer column keeps the integer's type, as Calcite defines it, and refuses by name
    on overflow. Declare a widened aggregate on the source — the Akade README shows accumulate —
    or cast the argument.
  • A confinement carried along a Related path, and one whose confining kind lives on a Through
    parent, are still refused by name.
  • The tutorial's recorded outputs lag the code in nine blocks — plan digests and node ids, not
    results — and will be re-recorded when the tutorial resumes.

Verified with full test suite: 1,539 planner tests, 5,487 integration tests, sample code (including 94 Akade tests).

Full Changelog: v0.1...v0.2

Licence and attribution

ChalkQL is licensed under the Apache License 2.0.

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.

ChalkQL is an independent project and is not affiliated with or endorsed by the Apache Software Foundation.

Assets 2
Loading