v0.1.8

Everything since v0.1.7. This release moves rsync.ai to the rsync.ai Source-Available License, and calls the product rsync.ai everywhere: the repository is now rsync-ai/rsync.ai, and the Docker containers, the Kubernetes release and the MCP server carry the name. It adds no features.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync.ai/v0.1.8/install.sh | bash

Re-running it on an existing install upgrades it to v0.1.8. The containers are now named rsync-ai-*; your data is in volumes, which keep their names, and the upgrade replaces each container under its new name.

Kubernetes (Helm): one command. It sizes the install to your cluster. For this release, pass the chart version:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync.ai/v0.1.8/install-k8s.sh | RSYNC_CHART_VERSION=0.1.8 bash

Without RSYNC_CHART_VERSION=0.1.8, the v0.1.8 script still installs chart 0.1.7. A follow-up change moves the script's default to 0.1.8.

You can also pull the chart directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.8.

License

• New license: the rsync.ai Source-Available License v1.0 (LICENSE), replacing the Elastic License 2.0 from this release on. Running rsync.ai for your own business, and for personal or non-commercial projects, stays free. Without a separate agreement you may not sell it, host it for others, embed it in a product you sell, build a competing product from it, or share copies except free of charge for non-commercial purposes.
• Releases up to and including v0.1.7 stay under the Elastic License 2.0. Its text is kept in LICENSES/.
• Contributions now need a signed CLA, once, in addition to the DCO sign-off.
• Every connector's license field carries the new license's identifier, LicenseRef-rsync.ai-SAL-1.0, instead of MIT or ELv2.

Named rsync.ai, not rsync

rsync is also the name of a file-copy tool that has existed for decades. The product is rsync.ai, and the names below now say so.

• The repository is github.com/rsync-ai/rsync.ai. A clone lands in rsync.ai/. GitHub redirects the old web, git and raw addresses, so an install command that names rsync-ai/rsync still works.
• Docker: the self-host containers are named rsync-ai-* (rsync-ai-postgres, rsync-ai-orchestrator, ...). Scripts that call docker exec or docker logs with an old name need the new one.
• Kubernetes: the default release and namespace are rsync-ai, so resources are named rsync-ai-frontend, rsync-ai-secrets and so on. install-k8s.sh finds an install under the old default (rsync in rsync) and upgrades it in place. If you run helm yourself, keep upgrading under the name you installed with.
• AI clients: the setup snippets name the MCP server rsync-ai, and the server reports rsync.ai as its title. A client you already set up keeps working under the name you gave it.
• Load test: RSYNC_AI_EMAIL and RSYNC_AI_PASSWORD replace RSYNC_EMAIL and RSYNC_PASSWORD in scripts/loadtest.
• Settings (RSYNC_*), Kafka topics (rsync.*), the rsync-network Docker network and ai.rsync.* labels keep their names.

Removed

• The petstore example connector, a demo of a generated REST connector, and its OAuth provider entry.

Tested before release

This release changes names, wording and the license, not pipeline behaviour. It was checked by the test suites, not by a live install:

• The repository's CI on the release commit, and locally: the documentation and license guards, the installer tests, the Go packages and frontend tests the change touched.
• install-k8s.sh's in-place upgrade from the old default release name is covered by tests run against a real helm v3.16.4, not by an upgrade on a live cluster.
• A Docker upgrade from v0.1.7, which renames every container, was not run on a live install.

v0.1.7

Everything since v0.1.6. This release makes batch reloads and resumes correct, makes CDC upsert only on declared keys, lets you generate a connector from an OpenAPI document in the UI, and makes an upgrade run the new connector code.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.7/install.sh | bash

Re-running it on an existing install upgrades it to v0.1.7 and adds the keys for the new blob-staging store.

Kubernetes (Helm): one command. It sizes the install to your cluster. For this release, pass the chart version:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.7/install-k8s.sh | RSYNC_CHART_VERSION=0.1.7 bash

Without RSYNC_CHART_VERSION=0.1.7, the v0.1.7 script still installs chart 0.1.5. A follow-up change moves the script's default to 0.1.7.

You can also pull the chart directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.7.

Fixed from v0.1.6

• "Start with sample data" works on a Docker install. demo-warehouse now joins the connector network, so you no longer need the docker network connect workaround.

What changed

Batch pipelines

• Reload onto object storage (GCS, S3, Azure Blob, MinIO): if the old files cannot all be deleted, the run fails. Before, it loaded on top of them.
• Resume continues each table from where that table's own sweep stopped. A table whose row count is an exact multiple of the page size records that it finished.
• Completion: a run completes only after its post-run row check passes; a failed check fails the run. The next run re-sends batches the destination refused in an earlier run.
• MongoDB: a collection whose _id mixes types gets one column type per run.
• Stop also stops the pipeline's sink worker.
• Sink connector: it runs on every install, including Helm with connectors.cdc.enabled=false. A batch pipeline with a destination checks for it before it starts.
• MinIO down: a run stops staging through it after the first failure, instead of retrying on every batch.
• Removed: the STRICT_PREFLIGHT setting. It had no effect.

Change data capture

• Behaviour change: keyless tables. A keyless table into a PostgreSQL, MySQL or MongoDB destination is now a warning, not a blocker. On such a table, an UPDATE adds a row and a DELETE is dead-lettered.
• Declared keys only. The sink upserts only on a declared key, taken from the source's message key or the destination config. It never uses a guessed id column, which could merge distinct rows.
• Keyless MongoDB rows are keyed on their Kafka record, so a replay does not duplicate them. MongoDB changes within the same second keep their order.
• MySQL destination key columns are created indexable (VARCHAR), so an upsert matches its row instead of inserting it again on every replay.
• Reload sends its re-snapshot only once the connector reports it is running.
• Refused credentials: when a destination refuses the sink's credentials, the pipeline stops as needing configuration instead of retrying.
• SQL Server: removing a table from a pipeline deletes its topic. The sink no longer creates phantom topics when it restarts after an edit.
• CDC lag is reported in messages, from the newest reading.

Sink

• Destination outage: a worker holding rows through an outage is no longer restarted as stalled.
• One host per write: the first destination host that answers is final. A refused write is not sent again under another name for the same service. On Helm, the destination's real error is no longer replaced by no such host.
• Layout v2 files are always Parquet. Before, a connection saved with format infer (the default in the GCS and Azure Blob forms) got gzip JSON under a .parquet name.

Connectors

• Generate a connector from an OpenAPI document in the UI. Upload a file, paste it, or give a URL. Your browser fetches the URL, not the server.
• Generated connectors deploy. Their files are saved readable by the orchestrator, and each is deployed from its saved version.
• An upgrade runs the new connector code. The connector-seed image replaces each shipped connector version and keeps your generated connectors. A connector container is rebuilt when its code changes, not only when its version tag does.
• In-cluster databases default to no TLS. A database inside the same Kubernetes cluster (*.svc, *.cluster.local) is treated like a Docker service name. An in-cluster MongoDB or PostgreSQL connects from host, port, user and password, without a connection string. An explicit TLS or sslmode setting still wins.

Interface

• Pipeline pages say only what they measured.
  • A stage is marked done on its own evidence.
  • A CDC header reads "Setting up" during a Reload.
  • The header's row count and step agree with the table statistics and the timeline.
  • A status change shows within seconds.
• Table statistics refresh while a run is going and take a final reading when it ends. CDC health reads the current run's probe.
• Batch Checkpoints card: it stays readable with many tables, with a summary line, a filter and search, and one compact row per table.
• Stop closes the actions menu. A CDC pipeline no longer sticks at "Failed" after Stop and Resume.
• Chat finds a saved connection by its name, such as "Demo warehouse".
• The /chat heading reads "Data Pipeline".

Self-hosting

• Blob staging: blob passthrough stages through its own blob-staging MinIO, which connectors can reach. The platform MinIO stays off the connector network.
• Start-up: the orchestrator waits for the database migrations before it runs its boot queries.
• Alerts: a Postgres, Kafka or Kafka Connect that is still starting no longer raises a CRITICAL alert.
• LLM cost logging works.
• Installing from a checkout or a mirror: install.sh installs from a checkout (RSYNC_COMPOSE_DIR, with an explicit RSYNC_VERSION) or pulls images from a mirror (RSYNC_IMAGE_REGISTRY).

Kubernetes

• Blob staging: the chart ships a dedicated blob-staging MinIO that only connector pods may reach under networkPolicy.enabled. install-k8s.sh generates its keys.
• Generate is not available on Kubernetes yet. It refuses to save a connector and says why. Each pod keeps its own copy of the connector catalog, so a generated connector would not reach any other service.

Tested before release

Release candidates of this code were tested in three setups:

• A fresh VM. Installed with install.sh from a checkout. We generated a connector from an OpenAPI document and deployed it, and it passed its connection test. The sample-data demo worked. Batch statistics covered all 42 tables. A PostgreSQL CDC insert, update and delete landed, and the keyless-table warning showed.
• A v0.1.6 install upgraded in place. The upgrade replaced every shipped connector file with the new release's code. A connector container whose code had changed was rebuilt; on the next deploy, the unchanged image was reused.
• A fresh GKE cluster with Google Managed Kafka (mTLS). We ran a Helm fresh install and a Helm upgrade. Also tested:
  • PostgreSQL → GCS CDC, including Reload;
  • MongoDB → GCS batch, which wrote Parquet (300 of 300 rows);
  • the Generate refusal;
  • the blob-staging network path under network policy.

The GKE run found the last two connector and sink fixes above: in-cluster databases default to no TLS, and layout v2 files are always Parquet. Both are covered by tests, but neither was re-run on a live cluster.

v0.1.6

Everything since v0.1.5. This release makes three changes: a CDC Reload copies every PostgreSQL row, a default install carries four Kafka topics instead of 28, and the first-run demo runs with no language model configured.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.6/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.6/install-k8s.sh | bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.6.

Known issue: "Start with sample data" fails on a Docker install

On a Docker (Compose) install, the first-run demo fails with a 502 error: could not translate host name "demo-warehouse". The bundled demo database is not attached to the network the connectors run on. Kubernetes installs are not affected. To fix it, run this once on the host and then click Start with sample data again:

docker network connect --alias demo-warehouse rsync-ai-mcp rsync-demo-warehouse

The fix ships in the next release.

What changed

Change data capture

• Reload copies every row. A Reload or re-snapshot of a PostgreSQL CDC pipeline could lose rows. The bundled PostgreSQL CDC connector is patched so the snapshot reads the whole table.
• Start resumes a stopped connector every time, and incremental-snapshot signals work against a Kafka cluster that requires authentication.
• A pipeline's incremental-snapshot signals go to one topic with a fixed key.
• A PostgreSQL or MongoDB CDC pipeline no longer gets a schema-history topic or an empty DDL topic. Only the sources that use them get them: MySQL/MariaDB, SQL Server, Oracle and Db2.

Batch pipelines

• A column whose values mix types no longer loses rows on a columnar write.
• Resume re-reads batches the sink sent to the dead-letter topic, instead of skipping past them.
• A run whose rows did not all land now fails, instead of pausing on "Configure connections".
• Table statistics show the run that moved rows, not a later empty Resume.

Kafka topics

• Breaking: a default install provisions four platform topics instead of 28: domain.events, rsync.notifications, pii.scan.request and pii.scan.response. The unused agent command-bus topics are removed, and so are the planner's Kafka consumer, POST /api/v1/topology/topics/pipeline and ENABLE_KAFKA_CONSUMER. Topics an existing install already has are not deleted.
• domain.events keeps 7 days of messages instead of growing forever.
• Schema-drift topics are created only when RSYNC_SCHEMA_DRIFT_ENABLED=true.
• Dead-letter topics are created up front. A message that cannot be dead-lettered is retried, not skipped.

First-run demo

• "Start with sample data" reaches a pipeline confirmation with no language model configured, and onboarding opens the chat with the request already sent.
• The demo no longer fails with "Missing connector: sample-data".

Interface

• A redesigned chat home: suggestions built from your own connections, your real pipelines, a loading skeleton, and a wide layout.

Self-hosting

• MinIO now pulls cgr.dev/chainguard/minio, pinned by digest, for both the server and the mc job, because the upstream images no longer serve anonymous pulls. Helm chart 0.1.5 and older name the withdrawn images; docs/deployment/kubernetes.md shows the override.

Kubernetes

• The Postgres liveness probes name the database. The generation tier receives its Redis settings.
• Claim-check storage resolves to the release's own object store. A large batch no longer counts its manifest twice.

Tested before release

• A fresh Kubernetes cluster with in-cluster Kafka ran PostgreSQL CDC, MongoDB batch and the sample-data demo.
• A fresh VM installed with install.sh ran PostgreSQL CDC: snapshot row counts matched the source (5,000 and 20,000 rows), and a live insert, update and delete each landed within seconds. That VM found the known issue above.

v0.1.5

Everything since v0.1.4: Kubernetes installs on a single 4-vCPU node, change data capture
drains partitions in parallel, and login resolves through an interface that fails closed.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.5/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.5/install-k8s.sh | bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.5.

What changed

Kubernetes

• A single 4-vCPU node installs. install-k8s.sh waits for the cluster's own DaemonSets
  before measuring free capacity — a node reads emptier than it is for about 25 seconds after
  it registers — counts unscheduled pods as room already owed, and adds a rung below lean
  that trims replica counts rather than leaving pods Pending.
• Kafka's log directory is a subdirectory of its volume rather than the volume root, so the
  broker no longer trips over lost+found on a fresh PersistentVolume.

Change data capture

• CDC data topics are created with min(3, brokers) partitions instead of whatever the broker
  happened to auto-create them at. Kafka Connect creates them through topic.creation.* before
  producing, so auto-create never gets a turn. KAFKA_CDC_TOPIC_PARTITIONS sets it wider.
• The sink drains those partitions in parallel flush lanes sharded on (topic, partition), so
  one slow table no longer stalls the others. RSYNC_SINK_FLUSH_LANES (default 4) sets the
  width; per-key ordering is unchanged.

Identity

• Login resolves through an interface and fails closed: a provider name this build does not
  have refuses every login, and now says so at boot instead of waiting for the first user who
  cannot get in. The password path stays the tested default.

Interface

• PII Management is reachable from the nav.
• The README opens with a screenshot of the approval gate.

Fixes

• The PII detection tier can store a scan result again, and a failed load stops reading as zero.
• A dev build left inside cmd/kafka-sink-worker/ is ignored again.

Dependencies

• Two Dependabot rounds: Go OpenTelemetry 1.45.0, plus frontend and Python pins.

---

Re-pointed 2026-09-23

This tag was moved from 7e1e907 to c7ff98c and everything at 0.1.5 was
rebuilt from it. Nothing had installed the first cut yet, so the three fixes
below were folded into this release rather than spending a 0.1.6 on them.

• The Kubernetes temporal-adapter waits for Kafka. createKafkaProducer
  failing is log.Fatalf with no retry, exactly like the orchestrator, but only
  the orchestrator was wired to wait. A fresh kind install of chart 0.1.5
  brought the adapter up after 4 restarts, every one Failed to create Kafka producer ... connection refused. If you installed the chart before this, that
  crash-loop is what you saw, and it resolved itself once the broker accepted.
• kafka-connect restarts on failure. It defaulted to RestartPolicy=no, so
  a single OOM kill stopped CDC capture permanently while every topic still read
  lag 0 and the pipeline still read running — a dead producer and a caught-up
  one are indistinguishable from lag alone. Its memory cap is now 2 GiB.
• sniStrict is back to false (Traefik's own default). true refuses any
  handshake whose SNI does not match a configured certificate, which makes
  https://<ip>/ unreachable — the only address an operator has before DNS
  points anywhere. The TLS floor, cipher suites, certificate validation and HSTS
  are unchanged.

The move also picks up the post-tag follow-up, so install-k8s.sh now defaults
to chart 0.1.5 instead of 0.1.4 and Chart.yaml matches the release.

install.sh, docker-compose.quickstart.yml and SHA256SUMS are byte-identical
to the first cut and were not re-uploaded; the published checksums still verify.
The chart digest changed from sha256:fdb2db2c… to sha256:f337e4cd….

If you pulled 0.1.5 before 2026-09-23, re-pull — a moved tag does not
re-pull on its own.

v0.1.4

Everything since v0.1.3: a one-command Kubernetes installer, a multi-arch chart, and a round of change-data-capture, security and interface fixes. Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.4/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.4/install-k8s.sh | RSYNC_CHART_VERSION=0.1.4 bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.4. install-k8s.sh at this tag still defaults to chart 0.1.3, so pass RSYNC_CHART_VERSION=0.1.4 as above; the next commit on main moves that default.

What changed

Kubernetes

• One-command installer (install-k8s.sh) that fits the cluster it lands on, waits for a
  slow image pull instead of failing, and keeps Kafka Connect credentials across a pod
  restart. The chart's default image tag is multi-arch (amd64 and arm64).

Change data capture

• Auto-pickup: a new source table joins a running "whole database" CDC pipeline.
• MongoDB: a stalled source now fails loudly instead of reporting healthy; the heartbeat
  topic is named with the key Debezium reads; editing a pipeline's tables updates
  collection.include.list; a standalone mongod is refused before start_sync; MongoDB
  CDC/streaming runs that carry an enabled mask_pii are refused.
• Fixes for wrong row counts, counters that reset on restart, and cooldowns that never blocked.
• The assessor warns about PostgreSQL tables without a primary key that sit outside a CDC pipeline.

Connections and storage

• A Scope step, server-level MySQL, MongoDB and ClickHouse connections, multi-database CDC and
  mirror mapping; namespace listing (GET /connections/:id/namespaces).
• Table discovery lists up to 5000 tables with totals.
• Object-storage layout v2 for GCS, S3 and Azure Blob (no pipeline id in the path).
• Deleting a pipeline closes six cleanup gaps and no longer un-owns destination data.

Security

• Plain-http OAuth token endpoints are refused for Kafka in all four runtimes.
• The connector deploy gate fails closed when ENVIRONMENT is unset; MongoDB URI aliases are masked.
• Compose Kafka Connect honours KAFKA_* security settings.

Interface

• Monitoring Overview shows freshness, backlog, Kafka lag and failures; the pipeline page has an
  Assessment tab; Activity replaces Trace and Live events; Data flow reads top-down.
• Explorer: a model page shows its lineage graph, its schedule in words and who runs it.
• Stage durations come from one formatter, so a single transition no longer reads as a retry.

Removed

• The bundled observability-backend stack. Telemetry still exports over OTLP to whichever
  collector you configure; the seeded sentinel_config keys are now backend-neutral
  (migration 100).

v0.1.3

Thirteen changes since v0.1.2, centred on making MongoDB → GCS CDC correct for
multi-collection pipelines, plus operator notifications.

Data correctness (CDC)

• #48 — a locked destination_namespace was silently discarded on three
  destinations. A pipeline that declared and locked a destination namespace
  had its CDC rows written into the connection's database/dataset/schema
  instead. Nothing downstream can detect that: the rows land, the counts match,
  the lag is zero, and they are in the wrong place. Two independent halves, both
  fixed — the sink's document-DB CDC branch never forwarded the namespace (while
  its own batch write and reload-drop always did), and mongodb / bigquery /
  redshift accepted the parameter and ignored it. The other ten destinations were
  censused; object stores stay deliberately silent because they key by the source
  table id. Both write lanes — CDC and batch — are now guarded by mutation-proven
  tests.
• #42 — the sink derived its topic prefix by hand and subscribed to topics
  nobody writes. Self-masking, because the pre-create step then created the
  bogus topics: the pipeline reported Running / Healthy / 100% while delivering
  zero rows.
• #45 — every bronze object was written uncompressed, and a compressed parquet
  was unreadable. Billed twice over (object storage plus bytes scanned) and the
  compressed path did not round-trip.
• #39 — MongoDB → GCS CDC fixes and failure notifications.

Notifications

• #43 — admin-configured Slack and SMTP delivery, with email controls, so a
  stopped or failing pipeline reaches the team instead of sitting silent.

Deployment and self-host

• #47 — publish arm64 images, fail fast on an unreachable database, bind dev
  ports to loopback.
• #41 — the one container that resolves connectors was the one with no
  connector mount, which made no pipeline creatable on a quickstart install.
• #40 — arm64 platform pin and MinIO environment-variable fixes.

Explorer, docs and dependencies

• #46 document browse mode for MongoDB connections · #44 README leads with
  "AI-native data pipeline platform"
• #33, #50, #51 — dependency bumps (actions, pip, npm).

Upgrading

Image tags move to v0.1.3. The sink worker and the mongodb / bigquery / redshift
connector images must be pulled for the namespace fix to take effect — a pipeline
that sets no destination namespace is unaffected by it.