v0.1.6

Everything since v0.1.5. This release makes three changes: a CDC Reload copies every PostgreSQL row, a default install carries four Kafka topics instead of 28, and the first-run demo runs with no language model configured.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.6/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.6/install-k8s.sh | bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.6.

Known issue: "Start with sample data" fails on a Docker install

On a Docker (Compose) install, the first-run demo fails with a 502 error: could not translate host name "demo-warehouse". The bundled demo database is not attached to the network the connectors run on. Kubernetes installs are not affected. To fix it, run this once on the host and then click Start with sample data again:

docker network connect --alias demo-warehouse rsync-ai-mcp rsync-demo-warehouse

The fix ships in the next release.

What changed

Change data capture

• Reload copies every row. A Reload or re-snapshot of a PostgreSQL CDC pipeline could lose rows. The bundled PostgreSQL CDC connector is patched so the snapshot reads the whole table.
• Start resumes a stopped connector every time, and incremental-snapshot signals work against a Kafka cluster that requires authentication.
• A pipeline's incremental-snapshot signals go to one topic with a fixed key.
• A PostgreSQL or MongoDB CDC pipeline no longer gets a schema-history topic or an empty DDL topic. Only the sources that use them get them: MySQL/MariaDB, SQL Server, Oracle and Db2.

Batch pipelines

• A column whose values mix types no longer loses rows on a columnar write.
• Resume re-reads batches the sink sent to the dead-letter topic, instead of skipping past them.
• A run whose rows did not all land now fails, instead of pausing on "Configure connections".
• Table statistics show the run that moved rows, not a later empty Resume.

Kafka topics

• Breaking: a default install provisions four platform topics instead of 28: domain.events, rsync.notifications, pii.scan.request and pii.scan.response. The unused agent command-bus topics are removed, and so are the planner's Kafka consumer, POST /api/v1/topology/topics/pipeline and ENABLE_KAFKA_CONSUMER. Topics an existing install already has are not deleted.
• domain.events keeps 7 days of messages instead of growing forever.
• Schema-drift topics are created only when RSYNC_SCHEMA_DRIFT_ENABLED=true.
• Dead-letter topics are created up front. A message that cannot be dead-lettered is retried, not skipped.

First-run demo

• "Start with sample data" reaches a pipeline confirmation with no language model configured, and onboarding opens the chat with the request already sent.
• The demo no longer fails with "Missing connector: sample-data".

Interface

• A redesigned chat home: suggestions built from your own connections, your real pipelines, a loading skeleton, and a wide layout.

Self-hosting

• MinIO now pulls cgr.dev/chainguard/minio, pinned by digest, for both the server and the mc job, because the upstream images no longer serve anonymous pulls. Helm chart 0.1.5 and older name the withdrawn images; docs/deployment/kubernetes.md shows the override.

Kubernetes

• The Postgres liveness probes name the database. The generation tier receives its Redis settings.
• Claim-check storage resolves to the release's own object store. A large batch no longer counts its manifest twice.

Tested before release

• A fresh Kubernetes cluster with in-cluster Kafka ran PostgreSQL CDC, MongoDB batch and the sample-data demo.
• A fresh VM installed with install.sh ran PostgreSQL CDC: snapshot row counts matched the source (5,000 and 20,000 rows), and a live insert, update and delete each landed within seconds. That VM found the known issue above.

v0.1.5

Everything since v0.1.4: Kubernetes installs on a single 4-vCPU node, change data capture
drains partitions in parallel, and login resolves through an interface that fails closed.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.5/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.5/install-k8s.sh | bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.5.

What changed

Kubernetes

• A single 4-vCPU node installs. install-k8s.sh waits for the cluster's own DaemonSets
  before measuring free capacity — a node reads emptier than it is for about 25 seconds after
  it registers — counts unscheduled pods as room already owed, and adds a rung below lean
  that trims replica counts rather than leaving pods Pending.
• Kafka's log directory is a subdirectory of its volume rather than the volume root, so the
  broker no longer trips over lost+found on a fresh PersistentVolume.

Change data capture

• CDC data topics are created with min(3, brokers) partitions instead of whatever the broker
  happened to auto-create them at. Kafka Connect creates them through topic.creation.* before
  producing, so auto-create never gets a turn. KAFKA_CDC_TOPIC_PARTITIONS sets it wider.
• The sink drains those partitions in parallel flush lanes sharded on (topic, partition), so
  one slow table no longer stalls the others. RSYNC_SINK_FLUSH_LANES (default 4) sets the
  width; per-key ordering is unchanged.

Identity

• Login resolves through an interface and fails closed: a provider name this build does not
  have refuses every login, and now says so at boot instead of waiting for the first user who
  cannot get in. The password path stays the tested default.

Interface

• PII Management is reachable from the nav.
• The README opens with a screenshot of the approval gate.

Fixes

• The PII detection tier can store a scan result again, and a failed load stops reading as zero.
• A dev build left inside cmd/kafka-sink-worker/ is ignored again.

Dependencies

• Two Dependabot rounds: Go OpenTelemetry 1.45.0, plus frontend and Python pins.

---

Re-pointed 2026-09-23

This tag was moved from 7e1e907 to c7ff98c and everything at 0.1.5 was
rebuilt from it. Nothing had installed the first cut yet, so the three fixes
below were folded into this release rather than spending a 0.1.6 on them.

• The Kubernetes temporal-adapter waits for Kafka. createKafkaProducer
  failing is log.Fatalf with no retry, exactly like the orchestrator, but only
  the orchestrator was wired to wait. A fresh kind install of chart 0.1.5
  brought the adapter up after 4 restarts, every one Failed to create Kafka producer ... connection refused. If you installed the chart before this, that
  crash-loop is what you saw, and it resolved itself once the broker accepted.
• kafka-connect restarts on failure. It defaulted to RestartPolicy=no, so
  a single OOM kill stopped CDC capture permanently while every topic still read
  lag 0 and the pipeline still read running — a dead producer and a caught-up
  one are indistinguishable from lag alone. Its memory cap is now 2 GiB.
• sniStrict is back to false (Traefik's own default). true refuses any
  handshake whose SNI does not match a configured certificate, which makes
  https://<ip>/ unreachable — the only address an operator has before DNS
  points anywhere. The TLS floor, cipher suites, certificate validation and HSTS
  are unchanged.

The move also picks up the post-tag follow-up, so install-k8s.sh now defaults
to chart 0.1.5 instead of 0.1.4 and Chart.yaml matches the release.

install.sh, docker-compose.quickstart.yml and SHA256SUMS are byte-identical
to the first cut and were not re-uploaded; the published checksums still verify.
The chart digest changed from sha256:fdb2db2c… to sha256:f337e4cd….

If you pulled 0.1.5 before 2026-09-23, re-pull — a moved tag does not
re-pull on its own.

v0.1.4

Everything since v0.1.3: a one-command Kubernetes installer, a multi-arch chart, and a round of change-data-capture, security and interface fixes. Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.4/install.sh | bash

Kubernetes (Helm) — one command; it sizes the install to your cluster:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.4/install-k8s.sh | RSYNC_CHART_VERSION=0.1.4 bash

The chart is also available directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.4. install-k8s.sh at this tag still defaults to chart 0.1.3, so pass RSYNC_CHART_VERSION=0.1.4 as above; the next commit on main moves that default.

What changed

Kubernetes

• One-command installer (install-k8s.sh) that fits the cluster it lands on, waits for a
  slow image pull instead of failing, and keeps Kafka Connect credentials across a pod
  restart. The chart's default image tag is multi-arch (amd64 and arm64).

Change data capture

• Auto-pickup: a new source table joins a running "whole database" CDC pipeline.
• MongoDB: a stalled source now fails loudly instead of reporting healthy; the heartbeat
  topic is named with the key Debezium reads; editing a pipeline's tables updates
  collection.include.list; a standalone mongod is refused before start_sync; MongoDB
  CDC/streaming runs that carry an enabled mask_pii are refused.
• Fixes for wrong row counts, counters that reset on restart, and cooldowns that never blocked.
• The assessor warns about PostgreSQL tables without a primary key that sit outside a CDC pipeline.

Connections and storage

• A Scope step, server-level MySQL, MongoDB and ClickHouse connections, multi-database CDC and
  mirror mapping; namespace listing (GET /connections/:id/namespaces).
• Table discovery lists up to 5000 tables with totals.
• Object-storage layout v2 for GCS, S3 and Azure Blob (no pipeline id in the path).
• Deleting a pipeline closes six cleanup gaps and no longer un-owns destination data.

Security

• Plain-http OAuth token endpoints are refused for Kafka in all four runtimes.
• The connector deploy gate fails closed when ENVIRONMENT is unset; MongoDB URI aliases are masked.
• Compose Kafka Connect honours KAFKA_* security settings.

Interface

• Monitoring Overview shows freshness, backlog, Kafka lag and failures; the pipeline page has an
  Assessment tab; Activity replaces Trace and Live events; Data flow reads top-down.
• Explorer: a model page shows its lineage graph, its schedule in words and who runs it.
• Stage durations come from one formatter, so a single transition no longer reads as a retry.

Removed

• The bundled observability-backend stack. Telemetry still exports over OTLP to whichever
  collector you configure; the seeded sentinel_config keys are now backend-neutral
  (migration 100).

v0.1.3

Thirteen changes since v0.1.2, centred on making MongoDB → GCS CDC correct for
multi-collection pipelines, plus operator notifications.

Data correctness (CDC)

• #48 — a locked destination_namespace was silently discarded on three
  destinations. A pipeline that declared and locked a destination namespace
  had its CDC rows written into the connection's database/dataset/schema
  instead. Nothing downstream can detect that: the rows land, the counts match,
  the lag is zero, and they are in the wrong place. Two independent halves, both
  fixed — the sink's document-DB CDC branch never forwarded the namespace (while
  its own batch write and reload-drop always did), and mongodb / bigquery /
  redshift accepted the parameter and ignored it. The other ten destinations were
  censused; object stores stay deliberately silent because they key by the source
  table id. Both write lanes — CDC and batch — are now guarded by mutation-proven
  tests.
• #42 — the sink derived its topic prefix by hand and subscribed to topics
  nobody writes. Self-masking, because the pre-create step then created the
  bogus topics: the pipeline reported Running / Healthy / 100% while delivering
  zero rows.
• #45 — every bronze object was written uncompressed, and a compressed parquet
  was unreadable. Billed twice over (object storage plus bytes scanned) and the
  compressed path did not round-trip.
• #39 — MongoDB → GCS CDC fixes and failure notifications.

Notifications

• #43 — admin-configured Slack and SMTP delivery, with email controls, so a
  stopped or failing pipeline reaches the team instead of sitting silent.

Deployment and self-host

• #47 — publish arm64 images, fail fast on an unreachable database, bind dev
  ports to loopback.
• #41 — the one container that resolves connectors was the one with no
  connector mount, which made no pipeline creatable on a quickstart install.
• #40 — arm64 platform pin and MinIO environment-variable fixes.

Explorer, docs and dependencies

• #46 document browse mode for MongoDB connections · #44 README leads with
  "AI-native data pipeline platform"
• #33, #50, #51 — dependency bumps (actions, pip, npm).

Upgrading

Image tags move to v0.1.3. The sink worker and the mongodb / bigquery / redshift
connector images must be pulled for the namespace fix to take effect — a pipeline
that sets no destination namespace is unaffected by it.

v0.1.2 — first public release

The first public release of rsync.ai — a self-hosted data platform for moving data
between databases, warehouses, object stores and APIs. You describe the job in a sentence,
an agent turns it into an explicit staged plan, it pauses to ask you when something is
ambiguous, and it runs on Temporal so a long sync survives a restart.

Source-available under the Elastic License 2.0:
run it, modify it, use it internally for free. You just cannot resell it as a hosted
service. There is no hosted offering — every install is yours.

Install

Docker — one command. Requires Docker. The installer prompts for an OpenAI API key,
generates every other secret, and starts the full stack on http://localhost:3000. If the
stack does not come up it says so and exits non-zero rather than printing a success banner
over a dead stack.

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.2/install.sh | RSYNC_REF=v0.1.2 bash

Both halves of that install come from this one tag: the compose file is fetched from
RSYNC_REF, and the image tag is derived from it, so the file and the containers it starts
are the same commit. The README one-liner tracks whichever release is current and resolves
to the same thing today.

Kubernetes. The chart is published to the registry, so no checkout is needed:

helm install rsync oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.2 \
  --namespace rsync --create-namespace \
  --set secrets.jwtSecret="$(openssl rand -base64 32)" \
  --set secrets.encryptionKey="$(openssl rand -base64 32)" \
  --set secrets.postgresPassword="$(openssl rand -hex 24)" \
  --set secrets.minioAccessKey="$(openssl rand -hex 16)" \
  --set secrets.minioSecretKey="$(openssl rand -base64 32)"

That is the evaluation footprint — in-chart Postgres, Redis, Kafka, MinIO and Temporal, one
replica each, no backups. The chart runs the same images as the compose stack and can point
at managed Postgres, Redis, Kafka and object storage instead.
Kubernetes guide

Save secrets.encryptionKey / ENCRYPTION_KEY before you store a credential. It
encrypts every stored connection. Reinstalling with a different key makes every saved
connection permanently undecryptable.

Try it without a credential of your own

The stack bundles a sample-data source and a throwaway demo-warehouse Postgres, so you
can build and run a real pipeline end to end on the first-run checklist.
Try it in 5 minutes

What ships

• 21 connectors, each its own versioned container so you can pin or upgrade one without
  touching the rest. Every one is a source, 17 are also destinations, and five support
  change data capture — PostgreSQL, MySQL, SQL Server, Oracle and MongoDB.
• Batch and CDC as peers. Batch loads for anything, plus Debezium-backed CDC on those
  five databases.
• Human-in-the-loop gates. When the source is ambiguous — which tables, which schema,
  which key — the run pauses instead of guessing.
• Durable execution. Stages run as Temporal workflows, so a multi-hour sync survives a
  restart, a redeploy or a crashed worker.
• The Data Explorer. Ask in English and get SQL back, or write it yourself; keep the
  useful ones as saved queries with versions and diffs, or as models that rebuild on a cron,
  an interval, or after a given pipeline finishes.
• Your keys. Credentials are encrypted at rest with a key you hold. Point the LLM at
  OpenAI or at a local Ollama.

What was verified for this release

Publish run	36 jobs, all success — 13 service images, 21 connector images, connector discovery, chart publish
Images the default compose starts	14 of 14 present at tag 0.1.2 and pullable anonymously from ghcr
Helm chart	oci://ghcr.io/rsync-ai/charts/rsync-ai tag list is ["0.1.2"]
Installer and compose file at this tag	both fetch HTTP 200; RSYNC_REF=v0.1.2 maps to image tag 0.1.2

Rough edges

• Kubernetes on a managed cluster has not been run end to end. The chart is installed
  and exercised on kind, including the external-Postgres path. EKS, GKE and AKS against
  real RDS, MSK and S3 are reviewed starting points, not verified recipes — expect to
  iterate on IAM and networking.
• Pin a tag rather than tracking main if you want reproducibility. On main the
  compose file changes with every commit while main images track the last publish, so the
  two halves move at different rates.
• Read the changelog before upgrading. Bugs and gaps are tracked as
  GitHub issues — that list is the register.

Requirements

Docker 24+ and Compose v2, or Kubernetes 1.25+ with Helm 3.8+. 8 GB RAM minimum, 16 GB
recommended. An OpenAI API key, or a self-hosted Ollama instance.

Documentation

Quick start ·
Self-hosting ·
Kubernetes ·
Connector reference ·
Architecture ·
All docs