v0.7.0

ChalkQL v0.7.0

ChalkQL 0.7.0 makes the entitlements explanation something a host can show as it comes. It also lets a schema keep names that SQL reserves, and gives the engine's clock to every way of running a statement.

• The explanation reads as SQL. ExplainAsync now writes its expressions over the column names the policy uses: MissionId IN ('KESTREL', 'SENTINEL') rather than SEARCH($1, Sarg[...]). A host no longer needs code to turn positions back into names.
• The explanation shows the mask that applies. The mask is a rule's own, else the column's, and none where the column is in full.
• Reserved words work as names. A column, table or schema named Open, At, Value, Year or national works unquoted. That includes the policy ChalkQL writes for it.
• @now follows the clock everywhere. On an engine with a clock, an unbound @now is the clock however the statement is run, including with a principal's context.

The explanation's text changes form, so this is a minor release. See Compatibility.

Targets .NET 10. A JDK 21 or newer is needed only to run the planner locally. Assemblies and namespaces retain the Chalk.* names.

NuGet packages:

Package	Licence
ChalkQL	Elastic License 2.0	engine, planner client, catalogue, execution, entitlements and standing views
ChalkQL.Sources	Elastic License 2.0	POCO, Akade.IndexedSet, ADO.NET, DuckDB and source conformance implementations

The explanation reads as SQL

EntitledEngine.ExplainAsync and EntitledQuery.ExplainAsync say what a principal's policy comes to without running anything. Their expressions were the planner's own form: columns by position, operators as calls, literals with their types. Hosts that showed an explanation parsed it back into SQL themselves. One host carried about a hundred lines to do it, and still could not name every position.

They are now ChalkQL SQL over the names the policy uses:

Before	After
=($1, 'KESTREL')	MissionId = 'KESTREL'
SEARCH($1, Sarg['KESTREL':VARCHAR CHARACTER SET "UTF-8", 'SENTINEL'…])	MissionId IN ('KESTREL', 'SENTINEL')
WHEN =($5, 2) THEN FULL; ELSE REDACTED	WHEN created_by = 2 THEN FULL; ELSE REDACTED
SUBSTRING($2, 1, 1)	SUBSTRING(first_name, 1, 1)
WHEN true THEN null:VARCHAR CHARACTER SET "UTF-8"	NULL
OR(IS NOT NULL($5), IS NOT NULL($7))	EXISTS (SELECT 1 FROM threads WHERE threads.id = left_thread) OR EXISTS (…)
WHEN COALESCE(CASE WHEN … THEN 1 ELSE 2 END, 2) = 1 THEN FULL	WHEN activities.site_id = 1 THEN FULL
OR(AND(=($2, 'SECRET'), …), AND(…))	(Classification = 'SECRET' AND …) OR (Classification = 'SECRET' AND …)

• Names. A table's own columns are named as they are, and quoted where SQL reserves the name: "Open", and "User", never USER. Another table's column is table.column.
• Parents and paths. A row that reaches a parent it is entitled through, or a path's tables, is written as the EXISTS that says so. A rule a parent decides is written as the condition it is over the parent's columns, not as the code the child reads.
• Context. A context value left to execution, as in a plan prepared once for a shape, is @ctx.name, as a policy writes it.
• Grouping. A conjunction under a disjunction is written in parentheses.
• Every field. This covers RowPredicate, Residual, each column's Disclosure conditions, Mask and Placeholder, and a path's EndpointPredicate and PathPredicate.

The expressions are the folded ones the plan enforces, unchanged; only how they are written is new. Anything the renderer cannot say is given in the planner's form, so an explanation never fails for it.

The explanation shows the mask that applies

ExplainedColumn.Mask gave the column's own mask, for a text column the policy's default '********', whatever a rule declared. It gave it too where the principal sees the column in full. It now gives the mask the statement applies:

• Its source: a rule's own mask, such as 'WITHHELD' or a lambda v: template filled in for the column; else the column's.
• Its shape: one mask that applies wherever the column is masked is that mask. Masks that vary by row are WHEN … THEN <mask>; …; ELSE <mask>. Where nothing masks, Mask is empty.

Reserved words as names

SQL reserves a few hundred words, among them ordinary .NET property and database names: Open, Close, At, Value, Year, national. Such a name used to be a parse error wherever it was written unquoted. That included the policy ChalkQL writes: a tenancy column named At failed the engine's creation. Hosts renamed their properties to suit SQL. They no longer need to.

• Read as the name it is. A reserved word that names a schema, table or column is read as that name, in a statement and in a policy's conditions, masks and placeholders. Where a parse stops at one, ChalkQL quotes it in the catalog's spelling and parses again. Text that parsed before is untouched.
• Quoted where ChalkQL writes it. The policy compiler quotes the names it writes where they need it. That covers a tenancy column in the row predicate, a resource owner, and a column filled into a mask template. Every other policy keeps its text, and so its descriptor hash.
• Refused where it names nothing. A reserved word that names nothing is refused, saying so and how to quote it. The parser's list of expected tokens is cut to its first eight.
• Policy parse errors name what failed. A policy expression that does not parse is named in the message, with no line or column counted in text the host never wrote.
• Refused where it is ambiguous. An unquoted USER, CURRENT_DATE, CURRENT_TIMESTAMP, LOCALTIME or one of their kin is read as SQL's function, never as a column of the same name. Where such a column is in scope, the statement or condition is now refused as ambiguous rather than reading the function silently. Quote the name to read the column: WHERE "User" = 'ann'.

The list of reserved words ships with the package and is linked from the guide.

@now follows the clock on every path

On an engine that names a clock (ChalkEngineOptions.Clock = "now"), an unbound @now was the clock only when a statement was run with named values. Run with no values, with a list, on a host's ArenaPool, or with a RequestContext, it was refused: the statement has 1 '?' placeholders but 0 values were bound. So a plan prepared once for a context's shape could not read @now at all.

• Every path. An unbound @now is now the clock however the statement is run, as of the refresh whose tables it reads. Before any refresh sets the clock, the run is refused with ClockRequiredException, as one reading CURRENT_TIMESTAMP is.
• Lists. A list binds a named statement's parameters in the order they first appear. One value short, it leaves @now to the clock and fills the others in order. A full list binds @now as the host's value.
• Wrong counts. A list of the wrong length names the statement's parameters, rather than calling them '?' placeholders. An empty list is refused for a statement that has parameters.

A host that bound @now to the machine's time can leave it unbound. To keep the time moving while no data arrives, refresh the clock alone on a timer: engine.RefreshAsync(r => r.AdvanceClock(now)). A snapshot then reads its rows and its time from one refresh, and agrees with the standing views.

Changed

• The explanation's text is SQL over column names, and its mask is the one that applies.
• Reserved words are accepted as names, and the policy compiler quotes the names it writes where they need them.
• An unquoted niladic word that names a column in scope is refused as ambiguous.
• @now is bound from the clock on every run path.
• Messages: a wrong parameter count names the parameters, and a parse error's expected tokens are cut short.

Fixed

• Reserved tenancy columns: a tenancy column, resource owner or masked column named after a reserved word no longer fails the engine's creation.
• Unbound @now: a plan prepared for a context's shape can read @now from the engine's clock.

Documentation

• The Incremental README: its lists of what is maintained and what is refused now match the guide. Right, semi and anti joins, ORDER BY … LIMIT at the statement's end, index lookups over a range and bounded windows are maintained.
• TenancyPrincipal.CustomLists says what reads the lists: the host's own predicate and an access rule's When.
• The group-size floor is set through EntitlementsOptions.DefaultMinGroupSize.

Compatibility

• The explanation's text changes form. ExplainedTable, ExplainedColumn and ExplainedPath keep their members and types, but the text of their expressions is now SQL over column names. A host that parsed the old form should read the new one as it comes. One that compared it to a stored string should store the new one.

• Mask is now empty where a principal sees the column in full, and the rule's own mask where one applies.

• New refusals:

  • an unquoted USER, CURRENT_DATE or kin where a column of that name is in scope;
  • an empty parameter list for a statement that has parameters.

  Quote the name, or bind the parameters.

• Messages: a host that matched on the text of the parameter-count error or a parse error's expected tokens will see new wording.

• No change to the API surface, the protocol or the IR. A 0.6.3 planner serves a 0.7.0 client, and a 0.7.0 planner serves a 0.6.3 client. The explanation's text, and the accommodation of reserved words in statements and policies, are the planner's, so they follow the planner's version.

Full Changelog: v0.6.3...v0.7.0

Licence and attribution

ChalkQL is source available under the Elastic License 2.0 (ELv2).

Apache Calcite and its JVM dependencies are bundled into the planner artefact distributed with ChalkQL. Their licences and attribution are recorded in THIRD-PARTY-NOTICES.txt.