Skip to content

0.12.4

Latest

Choose a tag to compare

@ferenc-csaky ferenc-csaky released this 04 Nov 12:01
· 63 commits to main since this release
0.12.4
7ba7d36

Breaking Changes

  • Cost ties always picks Flink from Flink and PostgreSQL, with cost-model set to DEFAULT (#2557)
    • Before: the order of an internal map picked the engine, so the plan could change with enabled-engines or a leftover build/ folder
    • Now: Flink on every compile
    • Affected: deployed plans with such a table in PostgreSQL; a recompile moves it to Flink, and the new Flink job cannot restore the state of the old one

Important

Before deploying a recompiled plan of a running pipeline:

  1. Compare the Stage: line of each table in build/pipeline_explain.txt with the deployed build
  2. For a table that changed from postgres to flink, either keep it in PostgreSQL with /*+ engine(postgres) */, or deploy the new plan as a new Flink job without the old job's state

See Optimizer Configuration.

Features

  • Database tables and indexes in pipeline_explain for every table that a database stores (#2554)
    • Output: a --- physical --- part with one Table <name> (<engine>): section per database table, listing its primary key and selected indexes
    • Index lines: type and columns as in the index hint, such as index Orders_2_btree_c1c2 (btree): customerid, time; __pk_hash marked (generated hash key)
    • JSON: a database_tables array on each stored table in pipeline_explain.json
    • Changed format: the bare --- before each table's definition is now --- definition ---
    • Failed compiles: pipeline_explain.*, pipeline_source.sqrl and pipeline_mutation_database.json no longer written when a compile fails between DAG planning and the backward compatibility check
  • New function typesafe_classify in stdlib.typesafe, from Flink SQL Runner 0.12.4: classifies each row of an insert-only table with TypeSafe AI's Jev model (DataSQRL/flink-sql-runner#411)
    • Setup: IMPORT stdlib.typesafe.*;
    • Output: input columns, one typed <question>_answer column per question, _model, and _error with on_error => 'NULL'
    • Question kinds: noul, choice, score
    • API key: pipeline.global-job-parameters.typesafe.api-key in engines.flink.config, or the TYPESAFE_API_KEY environment variable
    • Details: module README
  • Agent skills on indexes and engine selection in implement-sqrl and configure-sqrl (#2554, #2557)
    • Index checks: the api-query-indexes guide reads a table's indexes from pipeline_explain.txt instead of deploy/plan/postgres.json
    • Engine selection: the cost-model values, the Flink pick on a tie, and the Stage: line that shows each table's engine

Fixes

  • Misplaced compile errors in DISTINCT, IMPORT and EXPORT statements, hints, and tables defined in other files (#2543)
    • Before: a DISTINCT error pointed into the query that the statement is rewritten into; an IMPORT error pointed at the start of the statement
    • Now: errors point at the column, package path or hint name they are about
    • Other files: Flink errors in a table from a .table.sql file or an imported script point into that file
    • Empty definitions: fail with Missing query or expression after ':=', at the table name
    • Output: a % in a message is kept; no Java stack trace printed with a Flink error

Maintenance

  • Flink SQL Runner 0.12.4, up from 0.12.3 (see its release notes) (7ba7d3614)
    • New function: typesafe_classify in the new stdlib-typesafe module
    • Late records after a restore: fixed in kafka-safe and upsert-kafka-safe with idle advancement on; needs Describe on the consumer group for fresh starts with 'scan.startup.mode' = 'group-offsets'
    • Watermark alignment: scan.watermark.alignment.* now applies with SOURCE_WATERMARK() in kafka-safe and upsert-kafka-safe
    • Zero idle timeout: 'scan.watermark.idle-timeout' = '0' with SOURCE_WATERMARK() disables idleness instead of failing
Dependency From To PR
Maven Enforcer Plugin 3.6.3 3.7.0 #2563
JaCoCo 0.8.15 0.8.16 #2561
Node.js (MCP Inspector test image) 26.12.1-alpine 26.13.0-alpine #2564

Full Changelog: 0.12.3...0.12.4