Repository navigation
Releases
Releases · daun-gatal/bezisync
v0.5.0
Added
- Partition by on Debezium - Iceberg tables on Debezium can now be partitioned per table in the Destination step, as on Flink CDC: a column, the year/month/day/hour of a date or timestamp, buckets of a hash or the leading part, up to four fields, plus Debezium's time of the change for change logs. Tables Partition by doesn't list keep the automatic partitioning (bucket by key, day of the change), compiled as before (a pipeline with no Route step keeps its configuration).
- Flink CDC reads Oracle - Oracle tables can be synced into StarRocks, Doris and Fluss, and into Iceberg on Flink CDC (Debezium stays Iceberg's default), with parallel first copies, added columns applied and restorable positions. It needs a Flink CDC image with the
oracleconnector (/opt/flink-cdc/connectors/oracle.jar; the reference image adds it). Flink CDC's Oracle reader is narrower than Debezium's, and preflight says so before a deploy: Enterprise, Standard or Express Edition up to 23ai only (flink_oracle_version; not Oracle Database Free or 26ai),ANALYZE ANYfor the first copy, which replaces the tables' optimizer statistics (flink_oracle_analyze), and, on a multitenant database, aLOG_MINING_FLUSHtable in the CDB root that BeziSync creates (flink_oracle_flush_table). Tables must have upper-case names; renaming or dropping a column stops the pipeline (Start over), and adding tables needs a Start over first (flink_tables_added). MongoDB, Vitess, TiDB and OceanBase stay on Debezium: Flink CDC has no pipeline source for them (ADR 0006). - More functions for Flink CDC computed columns - replace text, first and last characters, pad, join with a separator, first that isn't empty, absolute value, round (up, down, to digits), add, subtract, multiply, divide, year, month, day of the month, hour, date from text, days between, and as a decimal or floating-point number, picked by group (ADR 0008).
- Flink CDC expressions - in Expert mode a computed column, or a filter's condition, can be written as an expression in Flink CDC's functions (
CASE WHEN total > 100 THEN 'big' ELSE 'small' END), checked as you type (POST …/expressions:check). BeziSync parses it and writes its own rendering into the job: only documented functions (no user-defined, AI or regular-expression functions), no comments or;, and every function NULL-safe so no value can stop the job. Preflight'stransform_expressionchecks the columns and their kinds. - More Flink CDC destinations: Elasticsearch, Apache Paimon and OceanBase, and Kafka on Flink CDC - Elasticsearch (6, 7 or 8; one index per table, a document per row keyed by the primary key), Apache Paimon (primary-key tables in a filesystem catalog on S3 or an S3-compatible store) and OceanBase (MySQL tenants, through OceanBase Connector/J) run on Flink CDC, with connection checks that write and remove a test index, object or table. Kafka can now also run on Flink CDC (JSON without the schema, keyed by the primary key; Debezium stays its default), so a pipeline that needs Flink CDC can write Kafka. They need a Flink CDC image with the
kafka,elasticsearch,paimon+paimon-s3andoceanbase+oceanbase-clientJARs (the guide lists them; the reference image adds them). Paimon has no read-back: Events stay empty and health is Unknown. Hudi and MaxCompute aren't offered (ADR 0009). - Column types in the Destination table - the Transform step's Destination table (Flink CDC) shows each column's type: the source's type for a source column, and the kind a computed or added column writes (text, a whole number, …), and when a column is cut short.
- Primary keys from any written column (Flink CDC) - the Primary key picker offers every column the table writes, with its type: source columns under their new names, added fields and computed columns (ADR 0010). A computed key column must be text or a whole number, never empty and the same each time a row is written; BeziSync writes it NOT NULL, as destinations such as Iceberg require. On Debezium, which builds the message key before any step runs, added fields are shown but can't be picked.
Changed
- CORS
*no longer sends credentials - with*among the CORS origins, any site'sOriginused to be answered with credentials, so another site could read the API with a signed-in user's cookie. Origins listed by name keep credentialed CORS; any other origin*allows now gets a literal*without credentials, and the API warns at startup. A cross-origin UI that signs in with cookies must be listed by name; bearer tokens are unaffected. - Project policies that can't be read fail closed - stored policies that fail to parse used to fall back to the defaults, which allow everything. Now the fields that parsed are kept, the others take their strictest value, and every change in the project (saving a changed spec, creating, deploying, even with Deploy anyway) is refused with
policy_unreadableuntil an Admin fixes them: Settings → Policies lists what couldn't be read. Running pipelines keep running; pause, resume and delete still work. - Testing a runtime needs
runtime.manage- it calls the runtime and rewrites its test result and capacity; members who aren't Admins no longer see Test. MCPtest_runtimeis now a write tool, anddelete_pipelineasks forpipeline.deploy, the permission deleting already needed. - Labels are checked when an edit changes them - at most 20, slug keys, values of at most 63 characters, as on create (and manifest apply). Labels a pipeline already holds still save unchanged.
- Flink CDC pipelines refuse snapshot requests - Flink CDC can't copy one table again while the others stream (use Start over); a deploy that adds tables lets Flink CDC copy them itself instead of queuing a Debezium snapshot.
- The SSRF guard holds on every connection, not just the check - BeziSync used to check a host when testing a connection and then let the database or broker driver dial on its own, so discovery, preparing, cleanup, signals, canaries, probes and the Events tab could reach loopback or cloud-metadata addresses, and a DNS answer that changed after the check, a redirect or an address a server advertises (Kafka brokers, MongoDB members, NATS servers) was never checked. Every driver, HTTP client, emulator host and custom cloud endpoint now dials through the guard; SSH-tunnelled sources keep working. A connection whose host resolves to a refused address now fails on every path, not only its check: allow it with
BEZISYNC_SSRF_ALLOW_CIDRSif it is meant to be reached (an emulator onlocalhost, for example). An egress proxy set withHTTPS_PROXY/HTTP_PROXYis still used, after the target is checked, and now also for single sign-on, Iceberg and Pulsar admin calls, which ignored it before. - Upserted Iceberg tables are partitioned only by key columns - rows are upserted by writing deletes on the key, and Iceberg applies a delete only to rows in its own partition, so a table partitioned by a column outside its key (an
updated_atday, astatus) kept the old row whenever an update changed that column. Preflight now blocks it (iceberg_partition_key), and the editor shows such columns disabled; for time-like layout partition by the leading part of an increasing key (truncate[100000](id)). Flink CDC pipelines partitioned that way are blocked on their next deploy (they keep running); their tables may already hold duplicate rows: find them withSELECT <key>, count(*) FROM <table> GROUP BY <key> HAVING count(*) > 1, and recreate the table with Start over, dropping destinations. The same check covers existing tables partitioned off their own identifier fields, on both engines. - Partition checks use the type the engine writes - buckets of a true/false or floating-point value, truncating a UUID, two time transforms of one column, and columns written as lists or structures (on Debezium also PostgreSQL arrays, user-defined types and NUMERIC) are refused before the deploy instead of failing when the table is created. Decimals sent as double or text, binary sent as text and a time of day (a number on Debezium) are judged by what is written. Truncate widths go up to 1,000,000.
- Raw Debezium 1.9 options that load code or open connections are refused on Flink CDC - Flink CDC embeds Debezium 1.9, whose
source.debezium.database.history*,source.debezium.rac.nodesandsource.debezium.database.connection.adaptercould load a class, send the schema history elsewhere or hand the source's credentials to other hosts. A pipeline that set one fails preflight until it is removed. - Partition by is chosen in the Transform step - Iceberg's Partition by now has its own row in Transform, after the key, so the key, renames and computed columns chosen on the same screen are offered as they change; the Destination step says where it went and warns when a write-mode or engine change made a saved field invalid, and so does Review. Nothing is stored differently, and no pipeline restarts (ADR 0007).
- Flink CDC shows why Iceberg has no change log - "Every change appended" is shown unavailable on Flink CDC with the reason (its Iceberg writer always writes rows by the table's key), naming Debezium when it is a real choice. A pipeline switched to Flink CDC with that choice must pick another before continuing.
- Pipelines with Flink CDC computed columns, or a filter comparing a number for equality, restart once on upgrade to run the fixed rendering. Rolling back: an older BeziSync refuses a spec with an expression or one of the new functions; remove those steps first.
- StarRocks and Doris hosts must be a host name or an IP address - the host goes into the JDBC and load URLs, so a
/,?or&in it could add driver options. Connections whose host already is one are unchanged. - The preflight check reference names the engines - each source and destination section says...
v0.4.0
Added
- Custom Flink CDC clusters - in Expert mode, Review → Size → Custom sets how many readers a Flink CDC pipeline runs and what each one needs, and draws the TaskManagers they would be packed into; BeziSync packs them into TaskManagers that fit the runtime's nodes when it deploys (or takes the TaskManagers you set) and keeps that shape until the next deploy. Preflight blocks a cluster no node can hold, and says so when the runtime doesn't report its nodes. A deployed pipeline can be resized and restores its position; changing its number of readers needs Start over first, and preflight now says so instead of letting the restore fail.
- Flink CDC clusters shrink after their first copy - give each reader less while streaming and BeziSync repacks the cluster once every table is copied and a position is saved, restarting it from that position, then grows it back before copying again. A streaming shape that keeps failing goes back to the copy size and raises an alert; the
shrink_after_copy: suggestpolicy offers Shrink now instead. - Resource limits per project - Policies → Resources caps the CPU, memory and readers one pipeline may ask for at its peak, on both engines, and lists the running pipelines a new limit would block (they keep running). A custom Flink CDC cluster is now held to the largest size too, by the Flink CDC preset's totals; before, it passed whatever its size.
- Placement profiles - a Kubernetes runtime's Admins define where pipelines may run (node selector, tolerations, priority class, pod labels) and a pipeline picks one in Review. Preflight blocks a profile no node takes; a custom Flink CDC cluster is packed for the profile's nodes; a profile edit applies at each pipeline's next deploy, and one in use can't be removed.
- More Debezium settings, checked and suggested - queue and batch sizes, a queue memory limit, the reconnect wait, skipping unchanged updates, the position save interval and the log level. Preflight checks tuned settings exist in the image's Debezium release, snapshot threads fit the source's free connections, and the queue and copy fetch fit the heap; Review suggests settings from that evidence with one click to use them.
- Settings for each source and destination - PostgreSQL, MySQL/MariaDB, Oracle and MongoDB get their own streaming settings, and Kafka, Pub/Sub and Redis destinations their batching and retry settings, each offered only where it applies. MongoDB and Spanner refuse JDBC driver properties.
- Import and version changes know the settings - adopting a Debezium Server sorts every property into settings, wizard choices, what BeziSync manages, advanced and refused, creates typed settings (an out-of-bounds one is listed, never clamped) and keeps excluded and masked columns, which it used to drop. Changing the Debezium version shows what the release does to each pipeline's settings and refuses one it lacks.
- Setting limits per project - Admins bound any typed setting (a range, some choices, an always-set value, or hidden); Policies counts the pipelines a limit would block before saving, saves outside the limits are refused, older pipelines' next deploys are blocked, pinned values are written into each deploy, and Promote's plan names what a target project refuses.
- Flink CDC options - Engine settings offer the Flink CDC sources' and destinations' own options (connect timeouts and attempts, PostgreSQL's WAL release delay, StarRocks loads, Iceberg file size, the schema change timeout; experimental ones say so), and Experts can pass the connectors' pass-through options (JDBC driver, embedded Debezium, destination table properties) with the same safety rules as Debezium.
- Keep deleted rows - a Transform step that keeps every deleted row, marked deleted (
__deletedon Debezium,__opon Flink CDC), instead of removing it; the preview shows it, and a table that already has that column is refused. - Cut long text - a Transform step that cuts text columns to a length, on Debezium (PostgreSQL, MySQL) and Flink CDC; a masked column can't be cut, a non-text one is left as it is (and blocks on Flink CDC), and importing a Debezium Server moves its truncations into the step.
- Send tables elsewhere - a Transform step that routes the tables a pattern matches (
shop.orders_*) to one topic or table you name, on both engines, showing where every captured table goes; a table two routes match is refused, and preflight blocks tables of different shapes sent to one destination. - Computed columns on Flink CDC - a Transform step that adds a column from a function picked from a list (case, trim, length, join, part of text, a date as text, as text, as a number) over the table's columns, never typed as SQL; masked and left-out columns can't be inputs, preflight blocks an input that's gone or of another type, and Debezium says why it can't run it.
- Partition by for Iceberg on Flink CDC - the Destination step partitions each table by a column's value, year, month, day, hour, hash buckets or leading part; preflight blocks a partition column the rows don't carry, and a table that already exists partitioned otherwise (Iceberg keeps a table's partitioning).
- Settings for agents and CI - the MCP tool
list_engine_settings(andGET …/pipelines/{id}/settings) shows what a pipeline may tune with the project's limits applied;bezisync remote diffandapplyprint each setting change with what it does, each advanced property's class and what would break a running pipeline, and stop (exit 3) on a breaking change until--confirm. - Capacity checked when you deploy - preflight measures the runtime as it runs, checks memory as well as CPU, blocks a deploy the Kubernetes namespace's ResourceQuota would refuse, counts only what a redeploy adds, and shows the check as unknown when the runtime doesn't report its capacity. Both Helm charts (0.2.1) let BeziSync read ResourceQuotas.
- Custom sizes for Debezium pipelines - in Expert mode a pipeline can run at its own CPU (0.25 to 8) and memory (512 MiB to 16 GiB); BeziSync works out the Java heap so the JVM always keeps room. A project's largest size and a destination's smallest size apply to it by CPU and memory.
- What a deploy changes in the settings - Review's final checks, a deploy or rollback confirmation, the version diff and an approval request list each setting a deploy changes, from what to what, grouped by what it does to the running pipeline (restart, first copy, next copy, starts over), heaviest first.
- Engine settings in Review - Expert mode gets an Engine settings card: typed settings grouped by when they matter, each with its bounds, default and what changing it does, and advanced Debezium properties as one row per key, each checked by the server as you type (with the reason and where to set it instead when refused, or the offer to use the typed setting).
- Typed engine settings - a pipeline spec takes
tuning: settings from BeziSync's registry, each checked against the pipeline's engine, source and destination and its bounds, and carried by YAML, the API and MCP. The first ones: snapshot threads and the poll interval (Debezium), the connection pool (Flink CDC).GET /engineslists every setting with what changing it does to a running pipeline, and preflight and the version diff list the settings a deploy changes with that impact (settings_change). - A project policy for advanced Debezium properties - Admins decide whether pipelines may pass raw properties (Settings → Policies, Allow advanced properties); without it their deploy is blocked. Upgrading turns it on for every project that already has a pipeline using them, with an audit entry.
Changed
- Unknown checks - preflight checks BeziSync can't verify are now Unknown: grey, never shown as passed. Advanced properties that are allowed show as one, listing each key.
- Images - the
bezisyncandbezisync-webimages leave out the source tree'sREADME.mdfiles (the new per-directory contracts for contributors); nothing they run changes.
Fixed
- Review stops re-saving a change the server refused - when a save is refused (a property BeziSync doesn't allow, a setting only the server checks, someone else's edit, a network error), Review & deploy shows why and waits: it tries again when you change something, click Try again (for network and server errors) or leave the step, instead of saving every second. A refused advanced property is caught as you type, in Guided mode too, and one an older pipeline already has still saves.
- Review says which field to fix - "Not saved" now names the field (Not saved: fix Memory, or Memory and 2 more) and takes you to it, and each error sits under its own field: a custom memory over the project's limit is shown under Memory, not CPU, and a Flink CDC streaming size under the streaming field it is about. In Guided mode, an engine setting or advanced property that blocks saving is listed with Remove instead of blocking unseen.
- Keep deleted rows shows in the Transform step - added, it used to vanish from the Transform step while still saved (and shown in Review). It now sits under Event shape on Debezium and under Keep rows on Flink CDC, can be removed, and is offered once. Where it decides deletes, the Transform step, Review, the deploy's impact, the Overview and the Destination step say so instead of showing the row shape's or the destination's own choice, and a filter added after it on Flink CDC goes before it, as the server requires.
- A new Keep only some events filter saves - a filter added in the Transform step without column rules was refused on save ("match needs rules"), on both engines; it now saves.
- Advanced Debezium properties can't override BeziSync - a raw property may no longer change a key BeziSync sets for the pipeline (where events go, tables, columns and personal-data choices, the error p...
v0.3.0
Added
- Flink CDC engine - pipelines into Iceberg, StarRocks, Doris and Fluss can run on Flink CDC (a JobManager and TaskManagers per pipeline on Docker or Kubernetes), with parallel first copies, schema changes applied to the destination tables and saved positions kept by BeziSync. Debezium stays the engine for every other destination; each pipeline shows its engine. Transforms on Flink CDC are filters built from rules, renames and added fields; value formats, the outbox router, flattening and custom steps stay Debezium options.
- StarRocks, Apache Doris and Apache Fluss destinations - on Flink CDC: one primary-key table per source table, created and kept in step with the source. StarRocks and Doris are checked, read back in the Events tab and dropped on delete, and get no source column defaults (they refuse them); Fluss is checked for reachability and its tables are kept on delete. None of the three loads over TLS: keep them on a private network.
- Iceberg on Flink CDC - Iceberg pipelines from MySQL, MariaDB, PostgreSQL or SQL Server can choose Flink CDC in the wizard (Debezium stays the default) over the same REST and Glue catalog connections.
- Flink CDC settings in the wizard - the review step sets what schema changes and new columns do, how often the position is saved (and Iceberg commits), how fresh StarRocks and Doris loads are, and in Expert mode Flink settings from a reviewed list.
- Engine versions - Settings → Debezium versions is now Settings → Engine versions, with a Flink CDC tab. Flink CDC stays off until an Admin registers an image they build (BeziSync publishes none; the guide lists what it holds). MySQL/MariaDB sources need MySQL Connector/J added to it.
- Saved positions - a Flink CDC pipeline's Positions tab lists its saved positions (encrypted in the state database) and, while paused, restores one or starts over: copy every table again, read from now, or read since a time (MySQL, SQL Server).
- Flink CDC from scripts and agents - manifests take
spec.engine,flink_cdcsettings andflinkCdcImage;bezisync remote pipelines positions|restore|start-overand the ENGINE column inremote get; MCP toolslist_engines,list_saved_positions,restore_saved_position, andfrom_timeonreset_pipeline_offsets.
Changed
- Helm charts 0.2.0 - the control-plane and agent charts' Roles allow Services, which a Flink CDC pipeline's TaskManagers use to reach its JobManager on Kubernetes.
Install
- Images:
ghcr.io/daun-gatal/bezisync:0.3.0,ghcr.io/daun-gatal/bezisync-web:0.3.0 - Charts:
bezisync0.2.0,bezisync-agent0.2.0 (oci://ghcr.io/daun-gatal/charts) - CLI: the archives below (
checksums.txt;gh attestation verify <file> --owner daun-gatal)
v0.2.0
Added
- About BeziSync - Help & tour → About BeziSync shows the server version and the license (Apache 2.0), links the source code, and opens the licenses and notices of the third-party software in the web UI and the server.
- Licenses shipped with BeziSync - The web image serves
/licenses.txt; the server binary carries its own (bezisync licenses,GET /api/v1/system/licenses, andLICENSES.txtin each CLI archive). Each lists BeziSync's NOTICE and LICENSE, then every included dependency with its license text and notices.
Changed
- Lab testing policy - The open lab's testing policy now says BeziSync is open source (Apache 2.0) and links its GitHub repository, offers GitHub private vulnerability reporting alongside in-app Feedback, and points load testing at your own copy.
Install
- Images:
ghcr.io/daun-gatal/bezisync:0.2.0,ghcr.io/daun-gatal/bezisync-web:0.2.0 - Charts:
bezisync0.1.1,bezisync-agent0.1.1 (oci://ghcr.io/daun-gatal/charts) - CLI: the archives below (
checksums.txt;gh attestation verify <file> --owner daun-gatal)
v0.1.0
Added
- BeziSync is open source - the first public release under the Apache License 2.0: a self-service control plane for Debezium Server with preflight checks, source protection, one-click operations, evidence-based health, encrypted write-only secrets, RBAC and SSO, per-pipeline personal-data handling and AI agents over MCP
Install
- Images:
ghcr.io/daun-gatal/bezisync:0.1.0,ghcr.io/daun-gatal/bezisync-web:0.1.0 - Charts:
bezisync0.1.0,bezisync-agent0.1.0 (oci://ghcr.io/daun-gatal/charts) - CLI: the archives below (
checksums.txt;gh attestation verify <file> --owner daun-gatal)