v0.3

ChalkQL v0.3

ChalkQL 0.3 broadens the boundary between SQL and host code: functions and in-process tables can expose composite types, delegates can use familiar CLR types directly, and STRING and BINARY values can flow from Arrow batches into functions and aggregates as borrowed spans.

Numeric semantics have also been tightened so DECIMAL rescaling, floating-point rounding and casts agree with the databases ChalkQL pushes work to.

Targets .NET 10. A JDK 21 or newer is needed only to run the planner locally. Assemblies and namespaces retain the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Apache-2.0	engine, planner client, catalogue, execution and entitlements
ChalkQL.Sources	Apache-2.0	POCO, Akade.IndexedSet, ADO.NET, DuckDB and source conformance implementations

Highlights

Composite types

A host function may now return a record which SQL addresses by field:

public readonly record struct Classification(
    Utf8String Category,
    double Confidence);

SELECT id,
       classify_transaction(description, amount).category   AS category,
       classify_transaction(description, amount).confidence AS confidence
FROM transactions
WHERE classify_transaction(description, amount).confidence > 0.8

The same model applies to aggregates and in-process tables:

• a client-bodied scalar function or aggregate may return a record;
• a record-valued POCO or AkadeSource member becomes one composite column;
• SQL may filter, order and group using its scalar fields;
• a whole composite can travel through projection, subqueries, UNION ALL and outer joins;
• selected composites arrive as Arrow struct columns;
• GetComposite<T>, TryGetComposite<T> and ReadComposites<T> read them back into the host's own record type.

Composite types are deliberately one level deep. They have no whole-value ordering or equality; use their scalar fields where comparison, grouping or ordering is required.

Functions in familiar CLR types

Tier 1 functions and aggregates can use the same CLR types commonly exposed by POCO properties:

decimal, DateOnly, TimeOnly, DateTime, DateTimeOffset, TimeSpan and Guid.

These types can be used for parameters, results, composite fields and aggregate inputs or outputs without translating them into raw SQL representations first.

Text stays bytes

GetUtf8 now exposes a STRING value as a ReadOnlySpan<byte> over the Arrow batch's own buffer:

ReadOnlySpan<byte> symbol = batch.Column(0).GetUtf8(row);

STRING and BINARY arguments can pass into functions and aggregates the same way, avoiding a per-row decode or copy. Copy with ToUtf8String() or decode with GetUtf8String() only when a value needs to outlive the batch.

The same spans can also be used as alternate keys against appropriately configured Dictionary, HashSet and ConcurrentDictionary instances without allocating a temporary string.

Population-safe user aggregates

A user-defined aggregate may now declare .Population(), asserting that its result describes the population rather than revealing an individual row.

Population-only entitlement rules may allow such aggregates, including composite-valued results, while the existing group-size floor continues to guard small groups.

Repeated expressions are computed once

Within one execution step, repeated immutable or stable expressions are compiled once and computed once per batch.

The classifier in the example above therefore runs once per row even though its result is referenced several times. Built-ins, casts, CASE, IN and field access receive the same treatment.

Volatile() functions remain evaluated once per occurrence.

Sovereign zones by construction

Sources may declare a sovereign zone with Zone(...).

A catalogue represents one zone: mixing different declared zones, or mixing zoned and unzoned sources, is refused when the engine is created. A host serving several sovereign zones uses a separate engine and catalogue for each and chooses the appropriate one per request.

Zone declarations are deliberately a guardrail rather than information-flow logic inside the query planner.

Numeric semantics

DECIMAL midpoint rescaling now rounds away from zero, matching PostgreSQL, DuckDB, SQL Server and SQLite and making a pushed expression agree with local execution.

This is a result-changing correction from 0.2, which used half-even rounding during rescaling:

Expression	Operand	0.2	0.3
CAST(x AS DECIMAL(18, 2))	DECIMAL 0.125	0.12	0.13
CAST(x AS DECIMAL(18, 2))	DECIMAL -0.125	-0.12	-0.13

Floating-point ROUND now operates on the exact binary value represented rather than a decimalised approximation. Consequently, the same printed digits need not behave like an exact DECIMAL:

Expression	Operand	0.2	0.3
ROUND(x, 2)	DOUBLE 655.925	655.93	655.92
ROUND(x, 2)	DOUBLE 2.675	2.68	2.67
CAST(x AS DECIMAL(18, 1))	DOUBLE 0.25	0.2	0.3

Conversions between DECIMAL and floating point have likewise been corrected to round from the value actually represented rather than passing through System.Decimal.

Changed

Borrowed text is now explicit

The old Chalk.ArrowUtf8Extensions.GetUtf8 API returned a Utf8String that could accidentally be retained beyond the lifetime of the batch it borrowed from.

In 0.3, borrowed values use ReadOnlySpan<byte>:

// 0.2
registry.AddScalar<Utf8String, long>("byte_length", s => s.Length);
Utf8String ticker = ((StringViewArray)batch.Column(0)).GetUtf8(row);

// 0.3
registry.AddScalar<ReadOnlySpan<byte>, long>("byte_length", s => s.Length);
ReadOnlySpan<byte> ticker = batch.Column(0).GetUtf8(row);

Utf8String kept = ticker.ToUtf8String();   // copy when it must outlive the batch

Tier 1 STRING and BINARY inputs likewise use ReadOnlySpan<byte> for borrowed values or string / byte[] when a copy is required.

Source row-level security trust is explicit

TrustSourceRowSecurity() has been renamed TrustSourceRowLevelSecurity(...).

Trust now requires the host to assert the security preconditions ChalkQL relies upon: that the connection identifies the principal, row-level security is enabled and forced on entitled tables, and the source policy admits exactly the rows required by the entitlement.

Incomplete assertions are refused.

PrepareOptions is now a record

PrepareOptions can be composed with with, and all prepare options now propagate correctly through entitled prepares.

Fixed

• Enforcement.PushdownRequired now influences costing so a plan that actually pushes the required row predicate is preferred when one exists.
• Enforcement.PushdownRequired no longer mistakes an unrelated pushed filter for the entitlement predicate itself.
• Changes to source capabilities, dialect, row-level-security trust or sovereign zone now strand prepared plans that depend on the old source contract.
• Catalogue refreshes that change cross-source join policy or cost profile now reach the planner correctly.
• COUNT(DISTINCT …) over a LIST is refused rather than producing an incorrect answer.
• LIST values are now consistently refused where ordering or equality would be required, including join and partition keys.
• Non-strict functions with a string? parameter now receive null for SQL NULL rather than an empty string.
• Utf8String.TextEquals no longer permits non-ASCII UTF-8 bytes to compare equal to unrelated UTF-16 code units.
• Planning errors involving casts around user-function arguments now retain useful source positions.

Known limits

• Composite types are one level deep; nested composites and lists inside composites are refused.
• Composite values have no whole-value ordering or equality.
• Composite columns currently originate only from in-process sources.
• Composite values cannot yet be SQL parameters or table-function columns, and SQL cannot construct one with ROW(...).
• DECIMAL values wider than 28 digits require a Tier 2 kernel when used by host functions.
• ArenaAggregateSpec, ArenaScope and ColumnWriter.Child remain experimental under CHALK001.
• ChalkQL remains read-only, and some correlated and LATERAL query shapes are still refused.

On the wire

The IR and catalogue gain the composite type, field access and population-aggregate flag. These are additive protocol changes: existing field numbers are unchanged and the IR version does not move.

A remotely deployed planner must use the 0.3 planner JAR when composites or population-safe aggregates are present. PlannerArtifact.CopyToAsync exports the artefact matching the installed client.

Packages

Package	Version	Depends on
ChalkQL	0.3.0
ChalkQL.Sources	0.3.0	exactly ChalkQL 0.3.0

Both target .NET 10. A local planner requires JDK 21 or newer; the matching planner JAR is embedded in Chalk.Client.dll.

Full Changelog: v0.2...v0.3

Licence and attribution

ChalkQL is licensed under the Apache License 2.0.

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.

ChalkQL is an independent project and is not affiliated with or endorsed by the Apache Software Foundation.