You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Partition by on Debezium - Iceberg tables on Debezium can now be partitioned per table in the Destination step, as on Flink CDC: a column, the year/month/day/hour of a date or timestamp, buckets of a hash or the leading part, up to four fields, plus Debezium's time of the change for change logs. Tables Partition by doesn't list keep the automatic partitioning (bucket by key, day of the change), compiled as before (a pipeline with no Route step keeps its configuration).
Flink CDC reads Oracle - Oracle tables can be synced into StarRocks, Doris and Fluss, and into Iceberg on Flink CDC (Debezium stays Iceberg's default), with parallel first copies, added columns applied and restorable positions. It needs a Flink CDC image with the oracle connector (/opt/flink-cdc/connectors/oracle.jar; the reference image adds it). Flink CDC's Oracle reader is narrower than Debezium's, and preflight says so before a deploy: Enterprise, Standard or Express Edition up to 23ai only (flink_oracle_version; not Oracle Database Free or 26ai), ANALYZE ANY for the first copy, which replaces the tables' optimizer statistics (flink_oracle_analyze), and, on a multitenant database, a LOG_MINING_FLUSH table in the CDB root that BeziSync creates (flink_oracle_flush_table). Tables must have upper-case names; renaming or dropping a column stops the pipeline (Start over), and adding tables needs a Start over first (flink_tables_added). MongoDB, Vitess, TiDB and OceanBase stay on Debezium: Flink CDC has no pipeline source for them (ADR 0006).
More functions for Flink CDC computed columns - replace text, first and last characters, pad, join with a separator, first that isn't empty, absolute value, round (up, down, to digits), add, subtract, multiply, divide, year, month, day of the month, hour, date from text, days between, and as a decimal or floating-point number, picked by group (ADR 0008).
Flink CDC expressions - in Expert mode a computed column, or a filter's condition, can be written as an expression in Flink CDC's functions (CASE WHEN total > 100 THEN 'big' ELSE 'small' END), checked as you type (POST …/expressions:check). BeziSync parses it and writes its own rendering into the job: only documented functions (no user-defined, AI or regular-expression functions), no comments or ;, and every function NULL-safe so no value can stop the job. Preflight's transform_expression checks the columns and their kinds.
More Flink CDC destinations: Elasticsearch, Apache Paimon and OceanBase, and Kafka on Flink CDC - Elasticsearch (6, 7 or 8; one index per table, a document per row keyed by the primary key), Apache Paimon (primary-key tables in a filesystem catalog on S3 or an S3-compatible store) and OceanBase (MySQL tenants, through OceanBase Connector/J) run on Flink CDC, with connection checks that write and remove a test index, object or table. Kafka can now also run on Flink CDC (JSON without the schema, keyed by the primary key; Debezium stays its default), so a pipeline that needs Flink CDC can write Kafka. They need a Flink CDC image with the kafka, elasticsearch, paimon + paimon-s3 and oceanbase + oceanbase-client JARs (the guide lists them; the reference image adds them). Paimon has no read-back: Events stay empty and health is Unknown. Hudi and MaxCompute aren't offered (ADR 0009).
Column types in the Destination table - the Transform step's Destination table (Flink CDC) shows each column's type: the source's type for a source column, and the kind a computed or added column writes (text, a whole number, …), and when a column is cut short.
Primary keys from any written column (Flink CDC) - the Primary key picker offers every column the table writes, with its type: source columns under their new names, added fields and computed columns (ADR 0010). A computed key column must be text or a whole number, never empty and the same each time a row is written; BeziSync writes it NOT NULL, as destinations such as Iceberg require. On Debezium, which builds the message key before any step runs, added fields are shown but can't be picked.
Changed
CORS * no longer sends credentials - with * among the CORS origins, any site's Origin used to be answered with credentials, so another site could read the API with a signed-in user's cookie. Origins listed by name keep credentialed CORS; any other origin * allows now gets a literal * without credentials, and the API warns at startup. A cross-origin UI that signs in with cookies must be listed by name; bearer tokens are unaffected.
Project policies that can't be read fail closed - stored policies that fail to parse used to fall back to the defaults, which allow everything. Now the fields that parsed are kept, the others take their strictest value, and every change in the project (saving a changed spec, creating, deploying, even with Deploy anyway) is refused with policy_unreadable until an Admin fixes them: Settings → Policies lists what couldn't be read. Running pipelines keep running; pause, resume and delete still work.
Testing a runtime needs runtime.manage - it calls the runtime and rewrites its test result and capacity; members who aren't Admins no longer see Test. MCP test_runtime is now a write tool, and delete_pipeline asks for pipeline.deploy, the permission deleting already needed.
Labels are checked when an edit changes them - at most 20, slug keys, values of at most 63 characters, as on create (and manifest apply). Labels a pipeline already holds still save unchanged.
Flink CDC pipelines refuse snapshot requests - Flink CDC can't copy one table again while the others stream (use Start over); a deploy that adds tables lets Flink CDC copy them itself instead of queuing a Debezium snapshot.
The SSRF guard holds on every connection, not just the check - BeziSync used to check a host when testing a connection and then let the database or broker driver dial on its own, so discovery, preparing, cleanup, signals, canaries, probes and the Events tab could reach loopback or cloud-metadata addresses, and a DNS answer that changed after the check, a redirect or an address a server advertises (Kafka brokers, MongoDB members, NATS servers) was never checked. Every driver, HTTP client, emulator host and custom cloud endpoint now dials through the guard; SSH-tunnelled sources keep working. A connection whose host resolves to a refused address now fails on every path, not only its check: allow it with BEZISYNC_SSRF_ALLOW_CIDRS if it is meant to be reached (an emulator on localhost, for example). An egress proxy set with HTTPS_PROXY/HTTP_PROXY is still used, after the target is checked, and now also for single sign-on, Iceberg and Pulsar admin calls, which ignored it before.
Upserted Iceberg tables are partitioned only by key columns - rows are upserted by writing deletes on the key, and Iceberg applies a delete only to rows in its own partition, so a table partitioned by a column outside its key (an updated_at day, a status) kept the old row whenever an update changed that column. Preflight now blocks it (iceberg_partition_key), and the editor shows such columns disabled; for time-like layout partition by the leading part of an increasing key (truncate[100000](id)). Flink CDC pipelines partitioned that way are blocked on their next deploy (they keep running); their tables may already hold duplicate rows: find them with SELECT <key>, count(*) FROM <table> GROUP BY <key> HAVING count(*) > 1, and recreate the table with Start over, dropping destinations. The same check covers existing tables partitioned off their own identifier fields, on both engines.
Partition checks use the type the engine writes - buckets of a true/false or floating-point value, truncating a UUID, two time transforms of one column, and columns written as lists or structures (on Debezium also PostgreSQL arrays, user-defined types and NUMERIC) are refused before the deploy instead of failing when the table is created. Decimals sent as double or text, binary sent as text and a time of day (a number on Debezium) are judged by what is written. Truncate widths go up to 1,000,000.
Raw Debezium 1.9 options that load code or open connections are refused on Flink CDC - Flink CDC embeds Debezium 1.9, whose source.debezium.database.history*, source.debezium.rac.nodes and source.debezium.database.connection.adapter could load a class, send the schema history elsewhere or hand the source's credentials to other hosts. A pipeline that set one fails preflight until it is removed.
Partition by is chosen in the Transform step - Iceberg's Partition by now has its own row in Transform, after the key, so the key, renames and computed columns chosen on the same screen are offered as they change; the Destination step says where it went and warns when a write-mode or engine change made a saved field invalid, and so does Review. Nothing is stored differently, and no pipeline restarts (ADR 0007).
Flink CDC shows why Iceberg has no change log - "Every change appended" is shown unavailable on Flink CDC with the reason (its Iceberg writer always writes rows by the table's key), naming Debezium when it is a real choice. A pipeline switched to Flink CDC with that choice must pick another before continuing.
Pipelines with Flink CDC computed columns, or a filter comparing a number for equality, restart once on upgrade to run the fixed rendering. Rolling back: an older BeziSync refuses a spec with an expression or one of the new functions; remove those steps first.
StarRocks and Doris hosts must be a host name or an IP address - the host goes into the JDBC and load URLs, so a /, ? or & in it could add driver options. Connections whose host already is one are unchanged.
The preflight check reference names the engines - each source and destination section says whether Debezium, Flink CDC or both read or write it, and the PostgreSQL, MySQL / MariaDB, SQL Server and Oracle sections link the Flink CDC checks that also run for them (replica identity, replication slots for the copy, server ids, the Oracle checks), which were listed only under Flink CDC.
A computed date as a key says how to partition by date - put the timestamp column itself in the key and partition it by its day under Partition by.
Fixed
Jobs a crashed controller left running are taken back - a restart, cleanup or notification interrupted by a controller crash runs again on the next leader; an interrupted snapshot is marked failed ("Interrupted…") rather than sent twice. Project deletion and runtime removal no longer stay blocked behind such a job.
The "Position moved backwards" alert can fire - the stored position was dropped on the tick after each probe, so a position going back was never noticed.
Delivery canaries keep running after one is lost - a canary that never arrived stopped canary checking for the pipeline; the alert now names when the lost one was written, and the next canary goes out on schedule.
Keep deleted rows before a field step on Iceberg - [Keep deleted rows, Rename] without an explicit Row after change ran the rename on the change envelope; the row shape now comes first. Only pipelines with that chain restart on upgrade.
Review deploys the version it checked - Deploy names the version the final checks ran on, so it never deploys an edit someone saved after.
A setting a project policy pins can't be switched - an On/Off setting pinned by policy stayed clickable in the settings editor.
Sign-in opens when saving your profile finds the session expired - changing Guided/Expert mode or notification settings after the session expired failed silently.
Slow Flink CDC helper copies on Docker - copying the helper was cut off after 30 seconds instead of its two-minute limit.
A data race in the remote-runtime agent - its report read the desired workloads while they were being replaced.
IAM-role credentials for SQS, SNS and Iceberg checks - their checks sent the AWS credential chain through the SSRF guard, which refuses instance and pod metadata addresses; only service calls are guarded now.
Vault answers are bounded and redirects aren't followed - an answer is read up to 4 MiB, and a redirect no longer carries the Vault token to another address.
Routed Debezium Iceberg tables get their automatic partitioning - the automatic partitioning (buckets of the key, the day of the change) was set under the table's name before its route, which the Iceberg build never looks up, so a table a Route step sends elsewhere was always created unpartitioned. It now follows the route; tables routed to one destination share it when they agree. Routed Debezium Iceberg pipelines restart once on upgrade; tables they create afterwards are partitioned, existing ones keep their partitioning.
Iceberg checks no longer pass when the catalog can't be read - an access or network error reading a destination table counted as "the table doesn't exist yet", so the existing-table partitioning check passed; it now warns.
Renamed or left-out key columns on Debezium Iceberg upserts - Debezium's Iceberg build keys a table only by key columns its rows carry under the same name, so a key column renamed or left out narrowed the table's identity and upserts replaced rows sharing the rest of the key. Preflight blocks it (iceberg_key_columns), and warns when no key column is left (the table is appended).
Flink CDC pipelines no longer run the delivery canary - Flink CDC captures only the pipeline's tables, so a canary row never arrived; the canary is now off for them and preflight says why.
Flink CDC pipelines keep $ in values as written - the Flink CDC helper filled $name and ${name} anywhere in the pipeline definition, so a literal such as 'US$5' in a transform reached Flink CDC changed, and a value containing $BZ_… could have had a secret written into it. Now only the placeholders BeziSync writes as whole values are filled. Pipelines restart only when their helper restarts; nothing else changes.
Connection settings can't name another secret - a setting (not a secret) containing ${, such as a user name of ${BZ_STATE_PASSWORD}, would have been filled in by Debezium Server or the Flink CDC helper and sent to that connection's server. Saving one is refused, and a pipeline using a connection saved with one before stops compiling (422 connection_config) until the setting is changed; its running workload is left as it is.
Flink CDC refuses an existing Iceberg table without a key - Flink CDC 3.7.0 writes every delete, and the old values of every update, into such a table as ordinary rows. Preflight now blocks it (iceberg_flink_keyless_table); a table the pipeline creates is unaffected.
Wizard steps no longer save an unchanged pipeline - with several Iceberg tables, the Destination and Review steps saved the recorded table keys in another order on every visit, and Transform and Tables saved when the spec carried value formats or keys next to Flink CDC settings.
Unticking a partitioned table - its Partition by and recorded key stayed in the spec, and the server refused the save.
Preflight fixes for Iceberg partitioning point at the Transform step, and RustFS's upsert check no longer tells Flink CDC pipelines to choose a change log.
Flink CDC computed columns no longer stop on an empty value - upper/lower case, trim, length and part of text threw on a NULL, which stopped the job on the first such row; they now give an empty value.
Flink CDC "Join" with a column and "As text" run - they compiled to CAST(… AS STRING), which Flink CDC 3.7.0 doesn't accept, so the pipeline never started.
Flink CDC filters compare numbers by value - "is", "is not", "is one of" with a number matched nothing on a BIGINT or DECIMAL column (Flink CDC compares Java objects). They now compare as numbers.
Preflight and the schema-drift alert check computed columns - a computed column whose input left the table was only caught when the job failed.
Kafka SASL passwords with a backslash - the JAAS line escaped only "; a \ in a user or password broke sign-in. Pipelines whose Kafka credentials hold a \ restart once.
Flink CDC key checks - a key column the table doesn't write (left out, or a step's column since removed or renamed) is now refused before deploy (transform_key_column_unknown), instead of failing the job; the left-out check no longer compares the renamed name, and Debezium's "falls back to the primary key" finding no longer appears on Flink CDC pipelines.
A Flink CDC key column that can be empty is now a warning (transform_key_column_nullable): Flink CDC writes it as an empty text, so the rows where it is empty share one key and replace each other.
Flink CDC key editor shows an expression column's type - it said "decided when it runs" while the reason beside it named the type; it now shows the kind the server's check reads.
A nullable key column next to the primary key no longer warns (transform_key_column_nullable) - with the whole primary key in the key, rows can't share a key.