0.12.3

Features

• Keyed the records that the pipeline exports into a PARTITIONED BY Kafka table the same way the server keys mutation records (#2527)
  • Records with the same key stay on one Kafka partition, in order
  • Records written without a key are still read, through the new scan.key-fields.read-from-value option of the default kafka-safe connector
  • With the plain kafka connector, these records are dropped as before 0.12.2, and the compiler warns about it
• Added "standalone": true for script.include entries whose project is deployed on its own (636802567)
  • That project owns its internal Kafka tables, so kafka.json lists their topics by name under includedTopics instead of topics
  • The deployment leaves creating and configuring these topics, including their partition count, to that project
  • Internal Iceberg tables are not covered yet
• Supported separate formats for the Kafka message key and value (#2529)
  • key.format and value.format from a connector template or WITH options now apply as given, instead of being overwritten by format
  • Options of format, such as flexible-json.timestamp-format.standard, only reach the side that uses that format
  • The compiler notes when both formats are equal, and warns when a mutation table uses a format other than json or flexible-json, since the server writes mutations as JSON
• Extended the agent skills with guidance on:
  • What the engine hint selects: engine(flink) only when a served query needs an index on a computed column, such as a text_search GIN index (#2515)
  • When a primary_key hint is needed: only when the compile fails with Could not determine primary key for table [...] (#2515)
  • PARTITIONED BY Kafka keys, which columns they can use, and keeping internal Kafka tables on kafka-safe (#2527, #2530)
  • Separate Kafka key and value formats, and standalone includes (#2529, 636802567)

Fixes

• Filled in the uuid and auth metadata columns before building the Kafka key of a mutation record (#2530)
  • Before, a PARTITIONED BY on such a column keyed every record {} and sent all of them to one partition. Upsert tables with such a column in their PRIMARY KEY had the same problem on their compacted topic
  • The key now contains every key column, so an omitted nullable column gives {"col":null}, the same key as an explicit null and as the one Flink writes
• Fixed EXPORT and INSERT INTO statements that write into a PARTITIONED BY Kafka table, which failed to compile (#2526)
  • The clause only declares the Kafka message key, but it reached the Flink table, and Flink's Kafka sink rejects partitioned tables
• Reported a PARTITIONED BY column that the server cannot fill in, such as METADATA FROM 'timestamp', with an error that names the column and lists the columns that can be used (#2530)

Documentation

• Documented:
  • Separate Kafka key and value formats (#2529)
  • Standalone includes and includedTopics in kafka.json (636802567)
  • Which columns PARTITIONED BY can use as the Kafka key, and that it also keys the records the pipeline exports (#2527, #2530)
  • What the engine hint selects and when a primary_key hint is needed (#2515)

Maintenance

• Updated Flink SQL Runner to 0.12.3 (see its release notes), which:
  • Adds scan.key-fields.read-from-value to kafka-safe
  • Fixes idle source-watermark advancement in kafka-safe and upsert-kafka-safe
  • Fixes value-format metadata columns, such as debezium-json's value.source.timestamp, reading as NULL in kafka-safe tables with key.fields
• Updated Jackson, Vert.x, Spring Framework, Reactor Core, Micrometer, Vert.x Maven Plugin, Maven Surefire Plugin, Maven Failsafe Plugin, and Node.js

---

Full Changelog: 0.12.2...0.12.3