v0.3

ChalkQL v0.3

ChalkQL 0.3 broadens the boundary between SQL and host code: functions and in-process tables can expose composite types, delegates can use familiar CLR types directly, and STRING and BINARY values can flow from Arrow batches into functions and aggregates as borrowed spans.

Numeric semantics have also been tightened so DECIMAL rescaling, floating-point rounding and casts agree with the databases ChalkQL pushes work to.

Targets .NET 10. A JDK 21 or newer is needed only to run the planner locally. Assemblies and namespaces retain the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Apache-2.0	engine, planner client, catalogue, execution and entitlements
ChalkQL.Sources	Apache-2.0	POCO, Akade.IndexedSet, ADO.NET, DuckDB and source conformance implementations

Highlights

Composite types

A host function may now return a record which SQL addresses by field:

public readonly record struct Classification(
    Utf8String Category,
    double Confidence);

SELECT id,
       classify_transaction(description, amount).category   AS category,
       classify_transaction(description, amount).confidence AS confidence
FROM transactions
WHERE classify_transaction(description, amount).confidence > 0.8

The same model applies to aggregates and in-process tables:

• a client-bodied scalar function or aggregate may return a record;
• a record-valued POCO or AkadeSource member becomes one composite column;
• SQL may filter, order and group using its scalar fields;
• a whole composite can travel through projection, subqueries, UNION ALL and outer joins;
• selected composites arrive as Arrow struct columns;
• GetComposite<T>, TryGetComposite<T> and ReadComposites<T> read them back into the host's own record type.

Composite types are deliberately one level deep. They have no whole-value ordering or equality; use their scalar fields where comparison, grouping or ordering is required.

Functions in familiar CLR types

Tier 1 functions and aggregates can use the same CLR types commonly exposed by POCO properties:

decimal, DateOnly, TimeOnly, DateTime, DateTimeOffset, TimeSpan and Guid.

These types can be used for parameters, results, composite fields and aggregate inputs or outputs without translating them into raw SQL representations first.

Text stays bytes

GetUtf8 now exposes a STRING value as a ReadOnlySpan<byte> over the Arrow batch's own buffer:

ReadOnlySpan<byte> symbol = batch.Column(0).GetUtf8(row);

STRING and BINARY arguments can pass into functions and aggregates the same way, avoiding a per-row decode or copy. Copy with ToUtf8String() or decode with GetUtf8String() only when a value needs to outlive the batch.

The same spans can also be used as alternate keys against appropriately configured Dictionary, HashSet and ConcurrentDictionary instances without allocating a temporary string.

Population-safe user aggregates

A user-defined aggregate may now declare .Population(), asserting that its result describes the population rather than revealing an individual row.

Population-only entitlement rules may allow such aggregates, including composite-valued results, while the existing group-size floor continues to guard small groups.

Repeated expressions are computed once

Within one execution step, repeated immutable or stable expressions are compiled once and computed once per batch.

The classifier in the example above therefore runs once per row even though its result is referenced several times. Built-ins, casts, CASE, IN and field access receive the same treatment.

Volatile() functions remain evaluated once per occurrence.

Sovereign zones by construction

Sources may declare a sovereign zone with Zone(...).

A catalogue represents one zone: mixing different declared zones, or mixing zoned and unzoned sources, is refused when the engine is created. A host serving several sovereign zones uses a separate engine and catalogue for each and chooses the appropriate one per request.

Zone declarations are deliberately a guardrail rather than information-flow logic inside the query planner.

Numeric semantics

DECIMAL midpoint rescaling now rounds away from zero, matching PostgreSQL, DuckDB, SQL Server and SQLite and making a pushed expression agree with local execution.

This is a result-changing correction from 0.2, which used half-even rounding during rescaling:

Expression	Operand	0.2	0.3
CAST(x AS DECIMAL(18, 2))	DECIMAL 0.125	0.12	0.13
CAST(x AS DECIMAL(18, 2))	DECIMAL -0.125	-0.12	-0.13

Floating-point ROUND now operates on the exact binary value represented rather than a decimalised approximation. Consequently, the same printed digits need not behave like an exact DECIMAL:

Expression	Operand	0.2	0.3
ROUND(x, 2)	DOUBLE 655.925	655.93	655.92
ROUND(x, 2)	DOUBLE 2.675	2.68	2.67
CAST(x AS DECIMAL(18, 1))	DOUBLE 0.25	0.2	0.3

Conversions between DECIMAL and floating point have likewise been corrected to round from the value actually represented rather than passing through System.Decimal.

Changed

Borrowed text is now explicit

The old Chalk.ArrowUtf8Extensions.GetUtf8 API returned a Utf8String that could accidentally be retained beyond the lifetime of the batch it borrowed from.

In 0.3, borrowed values use ReadOnlySpan<byte>:

// 0.2
registry.AddScalar<Utf8String, long>("byte_length", s => s.Length);
Utf8String ticker = ((StringViewArray)batch.Column(0)).GetUtf8(row);

// 0.3
registry.AddScalar<ReadOnlySpan<byte>, long>("byte_length", s => s.Length);
ReadOnlySpan<byte> ticker = batch.Column(0).GetUtf8(row);

Utf8String kept = ticker.ToUtf8String();   // copy when it must outlive the batch

Tier 1 STRING and BINARY inputs likewise use ReadOnlySpan<byte> for borrowed values or string / byte[] when a copy is required.

Source row-level security trust is explicit

TrustSourceRowSecurity() has been renamed TrustSourceRowLevelSecurity(...).

Trust now requires the host to assert the security preconditions ChalkQL relies upon: that the connection identifies the principal, row-level security is enabled and forced on entitled tables, and the source policy admits exactly the rows required by the entitlement.

Incomplete assertions are refused.

PrepareOptions is now a record

PrepareOptions can be composed with with, and all prepare options now propagate correctly through entitled prepares.

Fixed

• Enforcement.PushdownRequired now influences costing so a plan that actually pushes the required row predicate is preferred when one exists.
• Enforcement.PushdownRequired no longer mistakes an unrelated pushed filter for the entitlement predicate itself.
• Changes to source capabilities, dialect, row-level-security trust or sovereign zone now strand prepared plans that depend on the old source contract.
• Catalogue refreshes that change cross-source join policy or cost profile now reach the planner correctly.
• COUNT(DISTINCT …) over a LIST is refused rather than producing an incorrect answer.
• LIST values are now consistently refused where ordering or equality would be required, including join and partition keys.
• Non-strict functions with a string? parameter now receive null for SQL NULL rather than an empty string.
• Utf8String.TextEquals no longer permits non-ASCII UTF-8 bytes to compare equal to unrelated UTF-16 code units.
• Planning errors involving casts around user-function arguments now retain useful source positions.

Known limits

• Composite types are one level deep; nested composites and lists inside composites are refused.
• Composite values have no whole-value ordering or equality.
• Composite columns currently originate only from in-process sources.
• Composite values cannot yet be SQL parameters or table-function columns, and SQL cannot construct one with ROW(...).
• DECIMAL values wider than 28 digits require a Tier 2 kernel when used by host functions.
• ArenaAggregateSpec, ArenaScope and ColumnWriter.Child remain experimental under CHALK001.
• ChalkQL remains read-only, and some correlated and LATERAL query shapes are still refused.

On the wire

The IR and catalogue gain the composite type, field access and population-aggregate flag. These are additive protocol changes: existing field numbers are unchanged and the IR version does not move.

A remotely deployed planner must use the 0.3 planner JAR when composites or population-safe aggregates are present. PlannerArtifact.CopyToAsync exports the artefact matching the installed client.

Packages

Package	Version	Depends on
ChalkQL	0.3.0
ChalkQL.Sources	0.3.0	exactly ChalkQL 0.3.0

Both target .NET 10. A local planner requires JDK 21 or newer; the matching planner JAR is embedded in Chalk.Client.dll.

Full Changelog: v0.2...v0.3

Licence and attribution

ChalkQL is licensed under the Apache License 2.0.

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.

ChalkQL is an independent project and is not affiliated with or endorsed by the Apache Software Foundation.

v0.2

ChalkQL v0.2

The second public release. Three themes: the planner now plans for the rows a request will
actually pull, a source can be an Akade.IndexedSet with every index it carries, and a bound or
a value that is a parameter no longer costs the plan anything a literal would not.

Targets .NET 10. A JDK 21 or newer is needed only to the planner locally. Assemblies and namespaces keep the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Apache-2.0	engine, planner client, catalogue, execution and entitlements
ChalkQL.Sources	Apache-2.0	POCO, ADO.NET, DuckDB and source conformance implementations

Highlights

An Akade.IndexedSet is a table

Chalk.Sources.Akade, shipped in ChalkQL.Sources, publishes an
Akade.IndexedSet as an ordinary SQL table and
discovers the indexes it can serve faithfully, with no adapter written by the host:

var source = AkadeSource.From("purchases", purchases).Build();

• Hash and ordered indexes on one member or on a tuple of two to four, written either as a
  lambda tuple or as a named key method declared with CompoundIndex. A hash index reports its
  exact distinct key count to the planner.
• Key types and comparers. Integer, decimal and temporal keys are ordered access paths on the
  strength of the type; float, double, string and Utf8String become ordered once the
  comparer the index was built with is declared, ChalkComparers.For<T>() or
  StringComparer.Ordinal. A descending comparer registers a descending index, which serves
  ORDER BY … DESC without a sort.
• LIKE 'p%' is an index lookup. An ordered string index is sent the half-open range the
  prefix stands for; Akade's trie is a new PREFIX index kind and is sent the prefix itself.
• An ordered index is read backwards for a range with no upper bound, so
  ORDER BY ts DESC LIMIT 1 is one walk and one row rather than a sort of the table.
• Ordered lookups stream lazily as Akade produces them, under a per-row order guard that
  refuses an out-of-order row by name; a consumer that stops after one row pays for one row.
• Lifecycle. The one rule a host keeps, no mutation overlapping an execution or a refresh,
  and the three ways to keep it: in place between requests, swapping the set behind a delegate,
  or the transactional Append and Replace while requests are in flight. The package README
  opens with a table of what each Akade index becomes, and a benchmark measures ChalkQL's
  execution overhead over an Akade set against Akade's own calls.

Plans that fit the request

• A limit reaches the leaf that can stop early. A LIMIT above a scan or an index lookup,
  through a projection and a filter, gives the leaf a row goal: the planner costs the leaf for
  the rows the limit will pull rather than its whole output, so a statement that wants one row
  out of an ordered index seeks once. The goal travels to sources as ScanRequest.RowGoal and
  IndexLookupRequest.RowGoal, a hint never a bound: the fetch above stays authoritative. The
  in-memory source sizes its first batch by it and doubles from there.
• A bound above a UNION ALL or a partitioned table is copied into every branch, and a
  branch that is one remote source's subtree carries the copy away as that source's own
  ORDER BY … LIMIT.
• LIMIT ? and OFFSET ? plan and execute. The bound travels as the parameter it is and is
  read when the execution starts; a negative value, a NULL and a fraction are each refused by
  name before any row moves.
• A prepare may say what it expects its parameters to be worth.
  PrepareAsync(sql, parameterValueHints, …) takes the same container an execution binds from,
  sparse, with null meaning nothing said and DBNull.Value meaning SQL NULL. The planner
  estimates a predicate against a hinted parameter from the column's statistics instead of
  guessing, and a hinted LIMIT ? gives the leaf its row goal. A hint informs an estimate and
  never a truth: the plan's tree, its parameters and its rows are what they would have been, the
  values bound at execution need not match, and a hint's value reaches no log line, exception,
  plan text or digest. One saved filter, WHERE amount BETWEEN ? AND ? ORDER BY unit_price LIMIT ?,
  plans ordering-first for an interactive grid and predicate-first for a report export, from the
  hints alone.
• A parameterised bound is pushed to a source where a literal one is. The generated SQL
  carries a placeholder where the dialect puts the count — LIMIT ?, FETCH NEXT ? ROWS ONLY,
  TOP (?) — and the executor writes the value bound at execution into the text before the query
  is sent, one mechanism for every dialect. The guide's federation section shows how a host
  enables limit pushdown and what each source is then sent.

Entitlements

• A grant confined along another tenancy kind applies to a table that holds one kind on its own
  row and inherits the other along a declared path: a reviewer for one supplier, only in one
  warehouse, over positions that carry the warehouse and reach the supplier through their
  product. The confinement is decided above the join, over the two rows together; column rules,
  both binding modes, the explain and the reconciler follow. Such a grant was refused before.
• A refusal names its near miss. A confinement that would have to be carried along a
  Related path is refused with the model that comes closest and why it is not close enough.

Safe to log

• A redacted marker can say where its value came from. A literal that is, in type and value,
  one of the request context's bound values prints the name it was bound under beside its
  pseudonym, /*REDACTED-3f9a1c2e:DECIMAL @ctx.tenant_id*/, in the plan text and in the query
  text a source failure quotes. A parameter written as @name reads as @name again in the
  redacted statement and plan text rather than as the positional placeholder the planner saw.
  The structural hash and every pseudonym are unchanged.

Faster

• GROUP BY on two or more keys is two to three times faster: a scan under a computing
  projection reads only the columns the expressions name, the field trimmer no longer keeps a
  declared collation's columns alive, and compound aggregate keys take the fixed-width image path.
• A top-N's heap pass is priced, so a top-N over a hundred thousand rows is no longer costed
  like one over ten.

Fixed

• A LIMIT no longer claims its input's ordering upward, which could let a sort above it be
  removed although it decided the answer.
• A redacted plan text over a SEARCH — any IN list of two or more — no longer fails with an
  internal error.
• SELECT … LIMIT ? no longer fails inside the planner with an assertion where support belongs.
• A source whose dialect cannot write an OFFSET is no longer handed one; Calcite's SQL Server
  dialect spells a bound TOP (n) and dropped the offset beside it, returning the wrong rows.
• A UNION ALL under a parameterised bound with an OFFSET no longer fails planning.
• A hash index whose key a scan does not project in full is no longer offered as a lookup over a
  key prefix.
• A range's bounds are stated in the index's own key order, so a descending key column no longer
  has its > and < bounds the wrong way round.
• The Akade adapters call only the operations that honour the comparer an index was built with;
  Akade's one-sided range calls do not, and answered an index whose order is not the CLR default
  with no rows.

Changed

• An Akade source's table is named after the source unless TableName says otherwise. The
  previous default was the word data. A host that relied on that name must now name it.
• The Akade sample shows only what a host does: AkadeSource.From over the README's example
  sets. Its hand-written index adapters, which the package superseded, moved into the test kit.
• A remote query's parameters are listed in textual order, one per placeholder, which is what
  a bound at the front of a statement needs. No recorded plan moved.
• The public API of Chalk.Sources.Akade is tracked in PublicAPI.Shipped.txt and
  PublicAPI.Unshipped.txt like every other assembly.

On the wire

Every addition is additive and a plan that carries none of them is byte-identical to its 0.1
form: Read.row_goal, IndexLookup.row_goal, IndexLookup.reverse, Fetch.count_param,
Fetch.offset_param, TopN.count_param, TopN.offset_param, RemoteQuery.rendered_bounds,
IndexRange.prefix, IndexKind.PREFIX, Index.reversal, InheritedVisibility.path_predicate,
ExplainedPath.path_predicate, PlanRequest.parameter_hints, RedactSqlRequest.context. The
planner's configuration hash moves with the cost model, so a host cannot mistake two sidecars that
plan differently for two that plan alike.

Known limits

• A bound with an OFFSET beside it is not copied into UNION ALL branches; the bound stays
  local above the union.
• SUM over an integer column keeps the integer's type, as Calcite defines it, and refuses by name
  on ov...

Debut

Debut release of ChalkQL, a new federated SQL query engine for .NET with row- and column-level entitlements — include the .NET client, executor and sources, the Apache Calcite planner sidecar, the proto contract, the corpus, the samples, the tutorial and the guide.

Targets .NET 10. A JDK 21+ is required only when running the Calcite planner locally.

NuGet packages:

Package	Licence
ChalkQL	Apache-2.0	engine, planner client, catalogue, execution and entitlements
ChalkQL.Sources	Apache-2.0	POCO, ADO.NET, DuckDB and source conformance implementations

The public packages use the ChalkQL name; assemblies and .NET namespaces retain Chalk.*.

Verified with full test suite: Java build 1,378 tests, .NET 5,275 integration tests, sample code.

Full Changelog: https://github.com/markhammond/chalkql/commits/v0.1

Licence and attribution

ChalkQL is licensed under the Apache License 2.0.

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.

ChalkQL is an independent project and is not affiliated with or endorsed by the Apache Software Foundation.