Skip to content

0.12.5

Latest

Choose a tag to compare

@ferenc-csaky ferenc-csaky released this 17 Nov 19:58
· 113 commits to main since this release
0.12.5
09163b5

Breaking Changes

  • Compile fails on JWT claims without an auth provider in scripts with METADATA FROM 'auth.<claim>' (#2582)
    • Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with Invalid metadataType AUTH, and a mutation with a claim column stopped the server at start
    • Now: fails with The script reads JWT claims (...), but engines.vertx.config has no auth provider., listing every claim column
    • Provider: an auth block with at least one issuer, the deprecated jwtAuth, or the deprecated oauthConfig with oauth2Options.site
    • Checked: function arguments and mutation columns, also from inline imports and in functions that a custom GraphQL schema leaves out; not functions whose name starts with _, which are not exposed
  • Unknown metadata keys fail the compile on mutation table columns, and dotted uuid and timestamp keys also on function arguments (#2580)
    • Before: a column with an unknown key, such as the typo 'auht.sub', became a field of the mutation input, so a client could set a value meant to come from the JWT; 'uuid.<x>' and 'timestamp.<x>' compiled as 'uuid' and 'timestamp'
    • Now: only 'uuid', 'timestamp' and 'auth.<claim>'; one error lists every bad column of the table
    • Function arguments: 'uuid.<x>', 'timestamp.<x>' and an empty claim such as 'auth.' fail the compile, where every request failed before
    • Without FROM: the column name is the key, so a timestamp column declared METADATA gets a watermark, as with METADATA FROM 'timestamp'
  • Auth configs that accept unsigned tokens rejected for jwtAuth and oauthConfig (#2576)
    • Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted alg: none tokens
    • Now: sqrl compile fails when jwtAuth has no key that can verify signatures, such as without pubSecKeys, jwks or keyStore; the server does not start when none of the keys that oauthConfig discovers can
  • New PostgreSQL connection defaults in the server: prepared statements cached, pooled connections recycled every 10 minutes (#2598)
    • Before: pgConnectOptions.cachePreparedStatements was false, so every query was parsed and planned on each call; poolOptions.maxLifetime was 0, so behind a Kubernetes Service a replica added after the server started got no traffic until a restart
    • Now: true, with up to 256 statements per connection, and 600 seconds
    • Affected: PgBouncer in transaction pooling mode, which can't serve named prepared statements unless it is 1.21 or later with max_prepared_statements
    • Generic plans: if a query gets slower once PostgreSQL switches it to a generic plan, set plan_cache_mode = force_custom_plan on the server's database role instead of turning off the cache
  • Included project's subscriptions and exports left to its own deployment for script.include entries with "standalone": true (#2596)
    • Before: the including project planned the include's subscriptions and exports again, so both deployments wrote the same Kafka topics, each with its own partition count and topic config
    • Now: reads the subscription topics that the include's deployment exposes, listed under includedTopics in kafka.json; skips the include's EXPORT and INSERT statements into the log, its internal Kafka tables and external tables, with a notice
    • Scope: subscriptions of the script.main in the include's package file and of its inline imports; writes of every script it imports; without `script.ld behavior
    • New errors: a kafka topic template with ${sqrl:table-id}; a project whose Flink job would write no table
    • Affected: deployed plans with a standalone include; the recompiled Flink job no longer writes these topics

Important

Before recompiling a deployed project:

  1. Give each package file that compiles a script with METADATA FROM 'auth.<claim>' an auth provider, including test and local overlays; a deployment that mounts its own vertx-config.json needs it in the package file too, with ${...} placeholders for values that differ per environment
  2. Without a provider, start the name of a claim function that a custom GraphQL schema leaves out with _; /*+no_query*/ does not exempt it
  3. Replace metadata keys of mutation columns other than 'uuid', 'timestamp' and 'auth.<claim>', such as 'uuid.id' with 'uuid'
  4. Behind PgBouncer in transaction pooling mode, set pgConnectOptions.cachePreparedStatements to false in engines.vertx.config, unless it is 1.21 or later with max_prepared_statements
  5. With a standalone include, set script.main in the included project's package file, and deploy the included project first, since its deployment now creates and writes the topics under includedTopics

See Migrating from jwtAuth and oauthConfig to move to the auth block.

Features

a token passes only when its iss, aud and signature all match the same issuer

  • Keys: inline JWK keys, a jwksUri, or OIDC discovery from the issuer URL; fetched once at startup, so restart the server after a key rotation
  • Options: leeway and algorithms on auth or per issuer; resource and scopesSupported for MCP discovery, which advertises the issuers without inline keys whose issuer is a URL
  • Validation: at sqrl compile and again at server startup; auth together with jwtAuth or oauthConfig is an error, also across package files
  • Deprecated: jwtAuth and oauthConfig, which still work and log a warning
  • Details: Authentication Configuration
  • All errors of a script in one compile (#2544)
    • Output: every error with its position; a statement that fails to parse or plan is skipped, and the rest are still planned
    • Knock-on errors: not reported, such as a table or column not found because its definition failed; a failed USE skips the rest of its script
    • Details: Errors
  • Logs for large query results, failed queries and health checks in the server (#2589)
    • Large results: jdbcStatementLogConfig.largeResults warns when a database query returns at least rowCountThreshold rows (default 10000), before the server converts them to JSON; off by default
    • Output: GraphQL field path, row count, elapsed time and SQL, cut after 2000 characters; never parameter values
    • Failed queries: always an ERROR with the field path and SQL; also when rows can't be converted to JSON
    • Health checks: a WARN with the reason when /health first reports a check as failing, an INFO when it recovers
    • Details: Query Logging
  • --project-root option for add-func (#2594)
    • Usage: add-func -r my-project MyFunction adds the UDF to my-project/functions/; the path is relative to the mounted directory and must exist
    • Default: /workspace/functions/ without -r, as before
  • Agent skills on the auth block, query logging and the new compiler checks in configure-sqrl, design-api, implement-sqrl, implement-udf, manage-connector and test-sqrl (#2544, #2566, #2576, #2580, #2582, #2584, #2589, #2592, #2593, #2594, #2596, #2598)
    • Auth: the auth block, nested claim paths, and an auth provider in each package file that compiles a claim script
    • Server: query logging and the new PostgreSQL connection defaults
    • Compiler: all errors in one compile, the allowed metadata keys, and the unused-endpoint and wrapped-filter warnings, the latter in the api-query-indexes guide
    • Kafka: the -subscription topic next to a mutation table, and what a standalone include writes
    • Corrections: -r is relative to /workspace; unknown hint names only warn; Parquet options take the parquet. prefix; poisson_distribution and exponential_distribution are cumulative, the latter taking the mean

Fixes

  • SUBSCRIBE on a mutation table looping its records, for mutation tables on Kafka (#2566)
    • Before: a SUBSCRIBE function that read a mutation table directly made Flink copy the table into a Kafka table with the mutation's own topic, and read its own output in an endless loop
    • Now: that Kafka table gets <table-name>-subscription in place of ${sqrl:table-name} in the kafka template's topic, such as Events-subscription; all other topics keep their names
    • New errors: a kafka template with a fixed topic that gives the mutation topic again; an EXPORT to log.<name> or logger.<name> that lands on a mutation topic, such as EXPORT Orders TO log.Events
  • Nested JWT claims not found in METADATA FROM 'auth.<path>' (#2584)
    • Before: 'auth.org.id' looked for a top-level claim named org.id, so every request failed with Forbidden
    • Now: reads the field id of the object claim org, at any depth, also ending on an array; at each level the longest matching key wins, so claims with dots such as 'auth.https://example.com/org_id' still work
    • Not supported: array indexing such as 'auth.realm_access.roles.0', which reads as a missing claim
    • New error: a path with an empty segment, such as 'auth.org.' or 'auth..id', fails the compile
  • JBang UDFs not found with a cached JAR, with JBang scripts in more than one directory (#2586)
    • Before: jbang-udfs.jar was cached next to whichever script came first, and a compile that reused it failed with Could not find module for the UDFs
    • Now: cached in the deepest directory that contains all JBang scripts of the project; every UDF resolves, whether the JAR is built or reused
    • Included projects: one JAR each, deployed as jbang-udfs-<namespace>.jar; a warning when the project and included projects outside its folder deploy more than one JBang JAR, since their bundled dependencies can clash
    • New error: two projects that define different JBang UDFs with the same class name
  • No indexes for filters in subqueries of API queries (#2593)
    • Before: filters inside EXISTS (...) and IN (SELECT ...) played no part in index selection, so PostgreSQL scanned the subquery's table on every request
    • Now: the subquery's table gets an index for its filters; with only = comparisons to the outer query, as in IN, the compared outer column gets one too
    • Joins: a WHERE filter above a join with an expression in its ON condition now reaches the join, so that table gets an index
    • Recompiled plans: can gain indexes on subquery tables and lose those for <> and wrapped columns; compare pipeline_explain.txt
  • No warning for filters that an index can't serve, with the column wrapped in an expression (#2593)
    • Before: a column inside a function, such as LOWER(email) = :email, got no index and no warning; a column inside arithmetic, such as agency_id + 1 = :agency, got an index that PostgreSQL can't use
    • Now: no index, and a warning at the table function that names the filter and the column, and how to rewrite it
    • Still indexed: a cast that only changes precision or nullability, or converts between character types
    • Also: <> no longer gets a btree index; a table with an index hint gets no warning
    • Details: Automatic index selection
  • No warning for endpoints that a custom GraphQL schema leaves out (#2592)
    • Before: a query, relationship or subscription that no API version's schema used was still computed and written to PostgreSQL or Kafka, without a warning
    • Now: No GraphQL schema uses these endpoints, but they are still computed and written to the database or log: ...
    • To stop computing one: remove it or start its name with _; hint a table that has to stay /*+no_query*/
    • Not listed: endpoints on Iceberg; none in test, which uses the inferred schema by default

Maintenance

  • Flink SQL Runner 0.12.6, up from 0.12.4 (see the release notes of 0.12.5 and 0.12.6) (09163b5d8)
    • Paused readers after a failover: fixed for sources in a watermark alignment group (scan.watermark.alignment.group)
    • Late backlog after a start or restore: fixed in kafka-safe and upsert-kafka-safe with idle advancement on; a subtask is reported idle only while none of its partitions has produced a row and each is read to the end of its log
  • Releases also on GitHub Packages, as well as Maven Central (#2616)
    • Before: releases went only to Maven Central, which caps what the com.datasqrl namespace can publish per month
    • Now: each release also goes to https://maven.pkg.github.com/DataSQRL/sqrl, including the sqrl-cli and sqrl-server-vertx jars that Maven Central leaves out
    • Building from source: Flink SQL Runner releases missing from Maven Central resolve from https://maven.pkg.github.com/DataSQRL/flink-sql-runner, with a GitHub token that has read:packages under the server id github
Dependency From To PR
Vert.x 5.2.2 5.2.3 #2609
Netty Codec 4.2.19.Final 4.2.20.Final #2611
Guava 33.7.2-jre 33.8.0-jre #2612
JBang (datasqrl/cmd image) 0.142.0 0.143.0 #2613

Full Changelog: 0.12.4...0.12.5

Range: 0.12.4...0.12.5, 24 commits: 23 PRs and the direct runner bump 09163b5. Error messages, option names and defaults come from the PR diffs and docs, and all six docs anchors resolve on the live site.

Left out:

  • #2573 and #2574 (bumps of the actions/checkout and actions/setup-node GitHub Actions): CI only, and past releases leave out CI changes.
  • #2614 (the Release docs check, plus the SECURITY.md and compatibility table updates): CI and release housekeeping.
  • #2579 (cookie consent) and #2607 (SEO): changes to the docs website only.
  • #2594's AGENTS.md edits: these are agent instructions for this repository, not shipped skills.

Placement calls you may want to change:

  • I put #2598 (the Postgres defaults) under Breaking Changes because the new default breaks PgBouncer in transaction pooling mode. It could go under Features.
  • I put #2596 under Breaking Changes because it changes deployed Flink plans and adds compile errors, the same reasoning 0.12.4 used for the cost-tie change. It's labelled fix:.

Check before publishing:

  • Savepoint restore after #2596: the PR says an old savepoint may need allowNonRestoredState now that some Flink sinks are gone, but the author hadn't verified it. I left it out of the notes and the callout.
  • Where the artifacts are: when I checked, SQRL 0.12.5 wasn't on Maven Central or GitHub Packages yet. Runner 0.12.6 was on GitHub Packages but not on Central. Make sure the tag's deploy job finished, since the GitHub Packages entry says each release goes there.
  • Docs links: they use the trailing-slash URLs, because the site now redirects to them.

I didn't create the GitHub release or push anything. To publish, save the block to a file and run gh release create 0.12.5 --title 0.12.5 --notes-file .

result: Release notes for SQRL 0.12.5 (24 commits since 0.12.4) are printed above as Markdown ready to paste into the GitHub release, which I didn't create.

✻ Churned for 1m 28s · done 8:53 PM

❯ i'd rather not count the postgres defaults and the server auth behavior change breaking

I've moved both entries out of Breaking Changes. The Postgres defaults (#2598) are now a Features entry, and the PgBouncer note stays on that entry instead ofad "the server auth behavior change" as the unsigned-token rejection from #2576, which is now the first entry under Fixes. Breaking Changes keeps three entries: claims without an auth provider (#2582), unknown metadata keys (#2580) and standalone includes (#2596).

Breaking Changes

  • Compile fails on JWT claims without an auth provider in scripts with METADATA FROM 'auth.<claim>' (#2582
    • Before: compiled, and the server checked no token: endpoints without a claim served any caller, endpoints with one failed with Invalid metadataType AUTH, and a mutation with a claim column stopped the server at start
    • Now: fails with The script reads JWT claims (...), but engines.vertx.config has no auth provider., listing every claim column
    • Provider: an auth block with at least one issuer, the deprecated jwtAuth, or the deprecated oauthConfig with oauth2Options.site
    • Checked: function arguments and mutation columns, also from inline imports and in functions that a custom GraphQL schema leaves out; not functions whohich are not exposed
  • Unknown metadata keys fail the compile on mutation table columns, and dotted uuid and timestamp keys also on function arguments (#2580)
    • Before: a column with an unknown key, such as the typo 'auht.sub', became a field of the mutation input, so a client could set a value meant to come from the JWT; 'uuid.<x>' and 'timestamp.<x>' compiled as 'uuid' and 'timestamp'
    • Now: only 'uuid', 'timestamp' and 'auth.<claim>'; one error lists every bad column of the table
    • Function arguments: 'uuid.<x>', 'timestamp.<x>' and an empty claim such as 'auth.' fail the compile, where every request failed before
    • Without FROM: the column name is the key, so a timestamp column declared METADATA gets a watermark, as with METADATA FROM 'timestamp'
  • Included project's subscriptions and exports left to its own deployment for script.include entries with "standalone": true (#2596)
    • Before: the including project planned the include's subscriptions and exports again, so both deployments wrote the same Kafka topics, each with its own partition count and topic config
    • Now: reads the subscription topics that the include's deployment exposes, listed under includedTopics in kafka.json; skips the include's EXPORT and INSERT statements into the log, its internal Kafka tables and external tables, with a notice
    • Scope: subscriptions of the script.main in the include's package file and of its inline imports; writes of every script it imports; without script.main, a warning and the old behavior
    • New errors: a kafka topic template with ${sqrl:table-id}; a project whose Flink job would write no table
    • Affected: deployed plans with a standalone include; the recompiled Flink job no longer writes these topics

Important

Before recompiling a deployed project:

  1. Give each package file that compiles a script with METADATA FROM 'auth.<claim>' an auth provider, including test and local overlays; a deployment that ig.jsonneeds it in the package file too, with${...}placeholders for values that differ per environment without -r, as before
  • Agent skills on the auth block, query logging and the new compiler checks in configure-sqrl, design-api, implement-sqrl, implement-udf, manage-connector and test-sqrl (#2544, #2566, #2576, #2580, #2582, #2584, #2589, #2592, #2593, #2594, #2596, #2598)
    • Auth: the auth block, nested claim paths, and an auth provider in each package file that compiles a claim script
    • Server: query logging and the new PostgreSQL connection defaults
    • Compiler: all errors in one compile, the allowed metadata keys, and the unused-endpoint and wrapped-filter warnings, the latter in the api-query-indexes guide
    • Kafka: the -subscription topic next to a mutation table, and what a standalone include writes
    • Corrections: -r is relative to /workspace; unknown hint names only warn; Parquet options take the parquet. prefix; poisson_distribution and exponential_distribution are cumulative, the latter taking the mean

Fixes

  • Unsigned tokens accepted with jwtAuth or oauthConfig configs whose keys can't verify signatures (#2576)
    • Before: Vert.x dropped keys it could not load, and a JWT provider left with no keys accepted alg: none tokens
    • Now: sqrl compile fails when jwtAuth has no key that can verify signatures, such as without pubSecKeys, jwks or keyStore; the server does not start when none of the keys that oauthConfig discovers can
  • SUBSCRIBE on a mutation table looping its records, for mutation tables on Kafka (#2566)
    • Before: a SUBSCRIBE function that read a mutation table directly made Flink copy the table into a Kafka table with the mutation's own topic, and read its own output in an endless loop
    • Now: that Kafka table gets <table-name>-subscription in place of ${sqrl:table-name} in the kafka template's topic, such as Events-subscription; all other topics keep their names
    • New errors: a kafka template with a fixed topic that gives the mutation topic again; an EXPORT to log.<name> or logger.<name> that lands on a mutation topic, such as EXPORT Orders TO log.Events
  • Nested JWT claims not found in METADATA FROM 'auth.<path>' (#2584)
    • Before: 'auth.org.id' looked for a top-level claim named org.id, so every request failed with Forbidden
    • Now: reads the field id of the object claim org, at any depth, also ending on an array; at each level the longest matching key wins, so claims with dots such as 'auth.https://example.com/org_id' still work
    • Not supported: array indexing such as 'auth.realm_access.roles.0', which reads as a missing claim
    • New error: a path with an empty segment, such as 'auth.org.' or 'auth..id', fails the compile
  • JBang UDFs not found with a cached JAR, with JBang scripts in more than one directory (#2586)
    • Before: jbang-udfs.jar was cached next to whichever script came first, and a compile that reused it failed with Could not find module for the UDFs of the other directories
    • Now: cached in the deepest directory that contains all JBang scripts of the project; every UDF resolves, whether the JAR is built or reused
    • Included projects: one JAR each, deployed as jbang-udfs-<namespace>.jar; a warning when the project and included projects outside its folder deploy more than one JBang JAR, since their bundled dependencies can clash
    • New error: two projects that define different JBang UDFs with the same class name
  • No indexes for filters in subqueries of API queries (#2593)
    • Before: filters inside EXISTS (...) and IN (SELECT ...) played no part in index selection, so PostgreSQL scanned the subquery's table on every req
    • Now: the subquery's table gets an index for its filters; with only = comparisons to the outer query, as in IN, the compared outer column gets one too
    • Joins: a WHERE filter above a join with an expression in its ON condition now reaches the join, so that table gets an index
    • Recompiled plans: can gain indexes on subquery tables and lose those for <> and wrapped columns; compare pipeline_explain.txt
  • No warning for filters that an index can't serve, with the column wrapped in an expression (#2593)
    • Before: a column inside a function, such as LOWER(email) = :email, got no index and no warning; a column inside arithmetic, such as `agency_id + 1 =at PostgreSQL can't use
    • Now: no index, and a warning at the table function that names the filter and the column, and how to rewrite it
    • Still indexed: a cast that only changes precision or nullability, or converts between character types
    • Also: <> no longer gets a btree index; a table with an index hint gets no warning
    • Details: Automatic index selection
  • No warning for endpoints that a custom GraphQL schema leaves out (#2592)
    • Before: a query, relationship or subscription that no API version's schema used was still computed and written to PostgreSQL or Kafka, without a warning
    • Now: No GraphQL schema uses these endpoints, but they are still computed and written to the database or log: ...
    • To stop computing one: remove it or start its name with _; hint a table that has to stay /*+no_query*/
    • Not listed: endpoints on Iceberg; none in test, which uses the inferred schema by default

Maintenance

  • Flink SQL Runner 0.12.6, up from 0.12.4 (see the release notes of 0.12.5 and 0.12.6) (09163b5d8)
    • Paused readers after a failover: fixed for sources in a watermark alignment group (scan.watermark.alignment.group)
    • Late backlog after a start or restore: fixed in kafka-safe and upsert-kafka-safe with idle advancement on; a subtask is reported idle only while none of its partitions has produced a row and each is read to the end of its log
  • Releases also on GitHub Packages, as well as Maven Central (#2616)
    • Before: releases went only to Maven Central, which caps what the com.datasqrl namespace can publish per month
    • Now: each release also goes to https://maven.pkg.github.com/DataSQRL/sqrl, including the sqrl-cli and sqrl-server-vertx jars that Maven Central leaves out
    • Building from source: Flink SQL Runner releases missing from Maven Central resolve from https://maven.pkg.github.com/DataSQRL/flink-sql-runner, with a GitHub token that has read:packages under the server id github
Dependency From To PR
Vert.x 5.2.2 5.2.3 #2609
Netty Codec 4.2.19.Final 4.2.20.Final #2611
Guava 33.7.2-jre 33.8.0-jre #2612
JBang (datasqrl/cmd image) 0.142.0 0.143.0 #2613

Full Changelog: 0.12.4...0.12.5