Repository navigation
Breaking Changes
- Compile fails on JWT claims without an auth provider in scripts with
METADATA FROM 'auth.<claim>'(#2582)- Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with
Invalid metadataType AUTH, and a mutation with a claim column stopped the server at start - Now: fails with
The script reads JWT claims (...), but engines.vertx.config has no auth provider., listing every claim column - Provider: an
authblock with at least one issuer, the deprecatedjwtAuth, or the deprecatedoauthConfigwithoauth2Options.site - Checked: function arguments and mutation columns, also from inline imports and in functions that a custom GraphQL schema leaves out; not functions whose name starts with
_, which are not exposed
- Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with
- Unknown metadata keys fail the compile on mutation table columns, and dotted
uuidandtimestampkeys also on function arguments (#2580)- Before: a column with an unknown key, such as the typo
'auht.sub', became a field of the mutation input, so a client could set a value meant to come from the JWT;'uuid.<x>'and'timestamp.<x>'compiled as'uuid'and'timestamp' - Now: only
'uuid','timestamp'and'auth.<claim>'; one error lists every bad column of the table - Function arguments:
'uuid.<x>','timestamp.<x>'and an empty claim such as'auth.'fail the compile, where every request failed before - Without
FROM: the column name is the key, so atimestampcolumn declaredMETADATAgets a watermark, as withMETADATA FROM 'timestamp'
- Before: a column with an unknown key, such as the typo
- Auth configs that accept unsigned tokens rejected for
jwtAuthandoauthConfig(#2576)- Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted
alg: nonetokens - Now:
sqrl compilefails whenjwtAuthhas no key that can verify signatures, such as withoutpubSecKeys,jwksorkeyStore; the server does not start when none of the keys thatoauthConfigdiscovers can
- Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted
- New PostgreSQL connection defaults in the server: prepared statements cached, pooled connections recycled every 10 minutes (#2598)
- Before:
pgConnectOptions.cachePreparedStatementswasfalse, so every query was parsed and planned on each call;poolOptions.maxLifetimewas0, so behind a Kubernetes Service a replica added after the server started got no traffic until a restart - Now:
true, with up to256statements per connection, and600seconds - Affected: PgBouncer in transaction pooling mode, which can't serve named prepared statements unless it is
1.21or later withmax_prepared_statements - Generic plans: if a query gets slower once PostgreSQL switches it to a generic plan, set
plan_cache_mode = force_custom_planon the server's database role instead of turning off the cache
- Before:
- Included project's subscriptions and exports left to its own deployment for
script.includeentries with"standalone": true(#2596)- Before: the including project planned the include's subscriptions and exports again, so both deployments wrote the same Kafka topics, each with its own partition count and topic config
- Now: reads the subscription topics that the include's deployment exposes, listed under
includedTopicsinkafka.json; skips the include'sEXPORTandINSERTstatements into the log, its internal Kafka tables and external tables, with a notice - Scope: subscriptions of the
script.mainin the include's package file and of its inline imports; writes of every script it imports; without `script.ld behavior - New errors: a
kafkatopic template with${sqrl:table-id}; a project whose Flink job would write no table - Affected: deployed plans with a standalone include; the recompiled Flink job no longer writes these topics
Important
Before recompiling a deployed project:
- Give each package file that compiles a script with
METADATA FROM 'auth.<claim>'an auth provider, including test and local overlays; a deployment that mounts its ownvertx-config.jsonneeds it in the package file too, with${...}placeholders for values that differ per environment - Without a provider, start the name of a claim function that a custom GraphQL schema leaves out with
_;/*+no_query*/does not exempt it - Replace metadata keys of mutation columns other than
'uuid','timestamp'and'auth.<claim>', such as'uuid.id'with'uuid' - Behind PgBouncer in transaction pooling mode, set
pgConnectOptions.cachePreparedStatementstofalseinengines.vertx.config, unless it is1.21or later withmax_prepared_statements - With a
standaloneinclude, setscript.mainin the included project's package file, and deploy the included project first, since its deployment now creates and writes the topics underincludedTopics
See Migrating from jwtAuth and oauthConfig to move to the auth block.
Features
a token passes only when its iss, aud and signature all match the same issuer
- Keys: inline JWK
keys, ajwksUri, or OIDC discovery from theissuerURL; fetched once at startup, so restart the server after a key rotation - Options:
leewayandalgorithmsonauthor per issuer;resourceandscopesSupportedfor MCP discovery, which advertises the issuers without inlinekeyswhoseissueris a URL - Validation: at
sqrl compileand again at server startup;authtogether withjwtAuthoroauthConfigis an error, also across package files - Deprecated:
jwtAuthandoauthConfig, which still work and log a warning - Details: Authentication Configuration
- All errors of a script in one compile (#2544)
- Output: every error with its position; a statement that fails to parse or plan is skipped, and the rest are still planned
- Knock-on errors: not reported, such as a table or column not found because its definition failed; a failed
USEskips the rest of its script - Details: Errors
- Logs for large query results, failed queries and health checks in the server (#2589)
- Large results:
jdbcStatementLogConfig.largeResultswarns when a database query returns at leastrowCountThresholdrows (default10000), before the server converts them to JSON; off by default - Output: GraphQL field path, row count, elapsed time and SQL, cut after 2000 characters; never parameter values
- Failed queries: always an
ERRORwith the field path and SQL; also when rows can't be converted to JSON - Health checks: a
WARNwith the reason when/healthfirst reports a check as failing, anINFOwhen it recovers - Details: Query Logging
- Large results:
--project-rootoption foradd-func(#2594)- Usage:
add-func -r my-project MyFunctionadds the UDF tomy-project/functions/; the path is relative to the mounted directory and must exist - Default:
/workspace/functions/without-r, as before
- Usage:
- Agent skills on the
authblock, query logging and the new compiler checks inconfigure-sqrl,design-api,implement-sqrl,implement-udf,manage-connectorandtest-sqrl(#2544, #2566, #2576, #2580, #2582, #2584, #2589, #2592, #2593, #2594, #2596, #2598)- Auth: the
authblock, nested claim paths, and an auth provider in each package file that compiles a claim script - Server: query logging and the new PostgreSQL connection defaults
- Compiler: all errors in one compile, the allowed metadata keys, and the unused-endpoint and wrapped-filter warnings, the latter in the
api-query-indexesguide - Kafka: the
-subscriptiontopic next to a mutation table, and what a standalone include writes - Corrections:
-ris relative to/workspace; unknown hint names only warn; Parquet options take theparquet.prefix;poisson_distributionandexponential_distributionare cumulative, the latter taking the mean
- Auth: the
Fixes
SUBSCRIBEon a mutation table looping its records, for mutation tables on Kafka (#2566)- Before: a
SUBSCRIBEfunction that read a mutation table directly made Flink copy the table into a Kafka table with the mutation's own topic, and read its own output in an endless loop - Now: that Kafka table gets
<table-name>-subscriptionin place of${sqrl:table-name}in thekafkatemplate'stopic, such asEvents-subscription; all other topics keep their names - New errors: a
kafkatemplate with a fixedtopicthat gives the mutation topic again; anEXPORTtolog.<name>orlogger.<name>that lands on a mutation topic, such asEXPORT Orders TO log.Events
- Before: a
- Nested JWT claims not found in
METADATA FROM 'auth.<path>'(#2584)- Before:
'auth.org.id'looked for a top-level claim namedorg.id, so every request failed withForbidden - Now: reads the field
idof the object claimorg, at any depth, also ending on an array; at each level the longest matching key wins, so claims with dots such as'auth.https://example.com/org_id'still work - Not supported: array indexing such as
'auth.realm_access.roles.0', which reads as a missing claim - New error: a path with an empty segment, such as
'auth.org.'or'auth..id', fails the compile
- Before:
- JBang UDFs not found with a cached JAR, with JBang scripts in more than one directory (#2586)
- Before:
jbang-udfs.jarwas cached next to whichever script came first, and a compile that reused it failed withCould not find modulefor the UDFs - Now: cached in the deepest directory that contains all JBang scripts of the project; every UDF resolves, whether the JAR is built or reused
- Included projects: one JAR each, deployed as
jbang-udfs-<namespace>.jar; a warning when the project and included projects outside its folder deploy more than one JBang JAR, since their bundled dependencies can clash - New error: two projects that define different JBang UDFs with the same class name
- Before:
- No indexes for filters in subqueries of API queries (#2593)
- Before: filters inside
EXISTS (...)andIN (SELECT ...)played no part in index selection, so PostgreSQL scanned the subquery's table on every request - Now: the subquery's table gets an index for its filters; with only
=comparisons to the outer query, as inIN, the compared outer column gets one too - Joins: a
WHEREfilter above a join with an expression in itsONcondition now reaches the join, so that table gets an index - Recompiled plans: can gain indexes on subquery tables and lose those for
<>and wrapped columns; comparepipeline_explain.txt
- Before: filters inside
- No warning for filters that an index can't serve, with the column wrapped in an expression (#2593)
- Before: a column inside a function, such as
LOWER(email) = :email, got no index and no warning; a column inside arithmetic, such asagency_id + 1 = :agency, got an index that PostgreSQL can't use - Now: no index, and a warning at the table function that names the filter and the column, and how to rewrite it
- Still indexed: a cast that only changes precision or nullability, or converts between character types
- Also:
<>no longer gets a btree index; a table with anindexhint gets no warning - Details: Automatic index selection
- Before: a column inside a function, such as
- No warning for endpoints that a custom GraphQL schema leaves out (#2592)
- Before: a query, relationship or subscription that no API version's schema used was still computed and written to PostgreSQL or Kafka, without a warning
- Now:
No GraphQL schema uses these endpoints, but they are still computed and written to the database or log: ... - To stop computing one: remove it or start its name with
_; hint a table that has to stay/*+no_query*/ - Not listed: endpoints on Iceberg; none in
test, which uses the inferred schema by default
Maintenance
- Flink SQL Runner
0.12.6, up from0.12.4(see the release notes of 0.12.5 and 0.12.6) (09163b5d8)- Paused readers after a failover: fixed for sources in a watermark alignment group (
scan.watermark.alignment.group) - Late backlog after a start or restore: fixed in
kafka-safeandupsert-kafka-safewith idle advancement on; a subtask is reported idle only while none of its partitions has produced a row and each is read to the end of its log
- Paused readers after a failover: fixed for sources in a watermark alignment group (
- Releases also on GitHub Packages, as well as Maven Central (#2616)
- Before: releases went only to Maven Central, which caps what the
com.datasqrlnamespace can publish per month - Now: each release also goes to
https://maven.pkg.github.com/DataSQRL/sqrl, including thesqrl-cliandsqrl-server-vertxjars that Maven Central leaves out - Building from source: Flink SQL Runner releases missing from Maven Central resolve from
https://maven.pkg.github.com/DataSQRL/flink-sql-runner, with a GitHub token that hasread:packagesunder the server idgithub
- Before: releases went only to Maven Central, which caps what the
| Dependency | From | To | PR |
|---|---|---|---|
| Vert.x | 5.2.2 |
5.2.3 |
#2609 |
| Netty Codec | 4.2.19.Final |
4.2.20.Final |
#2611 |
| Guava | 33.7.2-jre |
33.8.0-jre |
#2612 |
JBang (datasqrl/cmd image) |
0.142.0 |
0.143.0 |
#2613 |
Full Changelog: 0.12.4...0.12.5
Range: 0.12.4...0.12.5, 24 commits: 23 PRs and the direct runner bump 09163b5. Error messages, option names and defaults come from the PR diffs and docs, and all six docs anchors resolve on the live site.
Left out:
- #2573 and #2574 (bumps of the actions/checkout and actions/setup-node GitHub Actions): CI only, and past releases leave out CI changes.
- #2614 (the Release docs check, plus the SECURITY.md and compatibility table updates): CI and release housekeeping.
- #2579 (cookie consent) and #2607 (SEO): changes to the docs website only.
- #2594's AGENTS.md edits: these are agent instructions for this repository, not shipped skills.
Placement calls you may want to change:
- I put #2598 (the Postgres defaults) under Breaking Changes because the new default breaks PgBouncer in transaction pooling mode. It could go under Features.
- I put #2596 under Breaking Changes because it changes deployed Flink plans and adds compile errors, the same reasoning 0.12.4 used for the cost-tie change. It's labelled fix:.
Check before publishing:
- Savepoint restore after #2596: the PR says an old savepoint may need allowNonRestoredState now that some Flink sinks are gone, but the author hadn't verified it. I left it out of the notes and the callout.
- Where the artifacts are: when I checked, SQRL 0.12.5 wasn't on Maven Central or GitHub Packages yet. Runner 0.12.6 was on GitHub Packages but not on Central. Make sure the tag's deploy job finished, since the GitHub Packages entry says each release goes there.
- Docs links: they use the trailing-slash URLs, because the site now redirects to them.
I didn't create the GitHub release or push anything. To publish, save the block to a file and run gh release create 0.12.5 --title 0.12.5 --notes-file .
result: Release notes for SQRL 0.12.5 (24 commits since 0.12.4) are printed above as Markdown ready to paste into the GitHub release, which I didn't create.
✻ Churned for 1m 28s · done 8:53 PM
❯ i'd rather not count the postgres defaults and the server auth behavior change breaking
I've moved both entries out of Breaking Changes. The Postgres defaults (#2598) are now a Features entry, and the PgBouncer note stays on that entry instead ofad "the server auth behavior change" as the unsigned-token rejection from #2576, which is now the first entry under Fixes. Breaking Changes keeps three entries: claims without an auth provider (#2582), unknown metadata keys (#2580) and standalone includes (#2596).
Breaking Changes
- Compile fails on JWT claims without an auth provider in scripts with
METADATA FROM 'auth.<claim>'(#2582- Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with
Invalid metadataType AUTH, and a mutation with a claim column stopped the server at start - Now: fails with
The script reads JWT claims (...), but engines.vertx.config has no auth provider., listing every claim column - Provider: an
authblock with at least one issuer, the deprecatedjwtAuth, or the deprecatedoauthConfigwithoauth2Options.site - Checked: function arguments and mutation columns, also from inline imports and in functions that a custom GraphQL schema leaves out; not functions whohich are not exposed
- Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with
- Unknown metadata keys fail the compile on mutation table columns, and dotted
uuidandtimestampkeys also on function arguments (#2580)- Before: a column with an unknown key, such as the typo
'auht.sub', became a field of the mutation input, so a client could set a value meant to come from the JWT;'uuid.<x>'and'timestamp.<x>'compiled as'uuid'and'timestamp' - Now: only
'uuid','timestamp'and'auth.<claim>'; one error lists every bad column of the table - Function arguments:
'uuid.<x>','timestamp.<x>'and an empty claim such as'auth.'fail the compile, where every request failed before - Without
FROM: the column name is the key, so atimestampcolumn declaredMETADATAgets a watermark, as withMETADATA FROM 'timestamp'
- Before: a column with an unknown key, such as the typo
- Included project's subscriptions and exports left to its own deployment for
script.includeentries with"standalone": true(#2596)- Before: the including project planned the include's subscriptions and exports again, so both deployments wrote the same Kafka topics, each with its own partition count and topic config
- Now: reads the subscription topics that the include's deployment exposes, listed under
includedTopicsinkafka.json; skips the include'sEXPORTandINSERTstatements into the log, its internal Kafka tables and external tables, with a notice - Scope: subscriptions of the
script.mainin the include's package file and of its inline imports; writes of every script it imports; withoutscript.main, a warning and the old behavior - New errors: a
kafkatopic template with${sqrl:table-id}; a project whose Flink job would write no table - Affected: deployed plans with a standalone include; the recompiled Flink job no longer writes these topics
Important
Before recompiling a deployed project:
- Give each package file that compiles a script with
METADATA FROM 'auth.<claim>'an auth provider, including test and local overlays; a deployment that ig.jsonneeds it in the package file too, with${...}placeholders for values that differ per environmentwithout-r, as before
- Agent skills on the
authblock, query logging and the new compiler checks inconfigure-sqrl,design-api,implement-sqrl,implement-udf,manage-connectorandtest-sqrl(#2544, #2566, #2576, #2580, #2582, #2584, #2589, #2592, #2593, #2594, #2596, #2598)- Auth: the
authblock, nested claim paths, and an auth provider in each package file that compiles a claim script - Server: query logging and the new PostgreSQL connection defaults
- Compiler: all errors in one compile, the allowed metadata keys, and the unused-endpoint and wrapped-filter warnings, the latter in the
api-query-indexesguide - Kafka: the
-subscriptiontopic next to a mutation table, and what a standalone include writes - Corrections:
-ris relative to/workspace; unknown hint names only warn; Parquet options take theparquet.prefix;poisson_distributionandexponential_distributionare cumulative, the latter taking the mean
- Auth: the
Fixes
- Unsigned tokens accepted with
jwtAuthoroauthConfigconfigs whose keys can't verify signatures (#2576)- Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted
alg: nonetokens - Now:
sqrl compilefails whenjwtAuthhas no key that can verify signatures, such as withoutpubSecKeys,jwksorkeyStore; the server does not start when none of the keys thatoauthConfigdiscovers can
- Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted
SUBSCRIBEon a mutation table looping its records, for mutation tables on Kafka (#2566)- Before: a
SUBSCRIBEfunction that read a mutation table directly made Flink copy the table into a Kafka table with the mutation's own topic, and read its own output in an endless loop - Now: that Kafka table gets
<table-name>-subscriptionin place of${sqrl:table-name}in thekafkatemplate'stopic, such asEvents-subscription; all other topics keep their names - New errors: a
kafkatemplate with a fixedtopicthat gives the mutation topic again; anEXPORTtolog.<name>orlogger.<name>that lands on a mutation topic, such asEXPORT Orders TO log.Events
- Before: a
- Nested JWT claims not found in
METADATA FROM 'auth.<path>'(#2584)- Before:
'auth.org.id'looked for a top-level claim namedorg.id, so every request failed withForbidden - Now: reads the field
idof the object claimorg, at any depth, also ending on an array; at each level the longest matching key wins, so claims with dots such as'auth.https://example.com/org_id'still work - Not supported: array indexing such as
'auth.realm_access.roles.0', which reads as a missing claim - New error: a path with an empty segment, such as
'auth.org.'or'auth..id', fails the compile
- Before:
- JBang UDFs not found with a cached JAR, with JBang scripts in more than one directory (#2586)
- Before:
jbang-udfs.jarwas cached next to whichever script came first, and a compile that reused it failed withCould not find modulefor the UDFs of the other directories - Now: cached in the deepest directory that contains all JBang scripts of the project; every UDF resolves, whether the JAR is built or reused
- Included projects: one JAR each, deployed as
jbang-udfs-<namespace>.jar; a warning when the project and included projects outside its folder deploy more than one JBang JAR, since their bundled dependencies can clash - New error: two projects that define different JBang UDFs with the same class name
- Before:
- No indexes for filters in subqueries of API queries (#2593)
- Before: filters inside
EXISTS (...)andIN (SELECT ...)played no part in index selection, so PostgreSQL scanned the subquery's table on every req - Now: the subquery's table gets an index for its filters; with only
=comparisons to the outer query, as inIN, the compared outer column gets one too - Joins: a
WHEREfilter above a join with an expression in itsONcondition now reaches the join, so that table gets an index - Recompiled plans: can gain indexes on subquery tables and lose those for
<>and wrapped columns; comparepipeline_explain.txt
- Before: filters inside
- No warning for filters that an index can't serve, with the column wrapped in an expression (#2593)
- Before: a column inside a function, such as
LOWER(email) = :email, got no index and no warning; a column inside arithmetic, such as `agency_id + 1 =at PostgreSQL can't use - Now: no index, and a warning at the table function that names the filter and the column, and how to rewrite it
- Still indexed: a cast that only changes precision or nullability, or converts between character types
- Also:
<>no longer gets a btree index; a table with anindexhint gets no warning - Details: Automatic index selection
- Before: a column inside a function, such as
- No warning for endpoints that a custom GraphQL schema leaves out (#2592)
- Before: a query, relationship or subscription that no API version's schema used was still computed and written to PostgreSQL or Kafka, without a warning
- Now:
No GraphQL schema uses these endpoints, but they are still computed and written to the database or log: ... - To stop computing one: remove it or start its name with
_; hint a table that has to stay/*+no_query*/ - Not listed: endpoints on Iceberg; none in
test, which uses the inferred schema by default
Maintenance
- Flink SQL Runner
0.12.6, up from0.12.4(see the release notes of 0.12.5 and 0.12.6) (09163b5d8)- Paused readers after a failover: fixed for sources in a watermark alignment group (
scan.watermark.alignment.group) - Late backlog after a start or restore: fixed in
kafka-safeandupsert-kafka-safewith idle advancement on; a subtask is reported idle only while none of its partitions has produced a row and each is read to the end of its log
- Paused readers after a failover: fixed for sources in a watermark alignment group (
- Releases also on GitHub Packages, as well as Maven Central (#2616)
- Before: releases went only to Maven Central, which caps what the
com.datasqrlnamespace can publish per month - Now: each release also goes to
https://maven.pkg.github.com/DataSQRL/sqrl, including thesqrl-cliandsqrl-server-vertxjars that Maven Central leaves out - Building from source: Flink SQL Runner releases missing from Maven Central resolve from
https://maven.pkg.github.com/DataSQRL/flink-sql-runner, with a GitHub token that hasread:packagesunder the server idgithub
- Before: releases went only to Maven Central, which caps what the
| Dependency | From | To | PR |
|---|---|---|---|
| Vert.x | 5.2.2 |
5.2.3 |
#2609 |
| Netty Codec | 4.2.19.Final |
4.2.20.Final |
#2611 |
| Guava | 33.7.2-jre |
33.8.0-jre |
#2612 |
JBang (datasqrl/cmd image) |
0.142.0 |
0.143.0 |
#2613 |
Full Changelog: 0.12.4...0.12.5