Skip to content

v0.1.7

Choose a tag to compare

View all tags
@rahulv8 rahulv8 released this 28 Sep 12:02
· 4 commits to main since this release
v0.1.7
11d3884
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Everything since v0.1.6. This release makes batch reloads and resumes correct, makes CDC upsert only on declared keys, lets you generate a connector from an OpenAPI document in the UI, and makes an upgrade run the new connector code.
Images are published for linux/amd64 and linux/arm64.

Install

Docker (Compose)

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.7/install.sh | bash

Re-running it on an existing install upgrades it to v0.1.7 and adds the keys for the new blob-staging store.

Kubernetes (Helm): one command. It sizes the install to your cluster. For this release, pass the chart version:

curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync/v0.1.7/install-k8s.sh | RSYNC_CHART_VERSION=0.1.7 bash

Without RSYNC_CHART_VERSION=0.1.7, the v0.1.7 script still installs chart 0.1.5. A follow-up change moves the script's default to 0.1.7.

You can also pull the chart directly: helm pull oci://ghcr.io/rsync-ai/charts/rsync-ai --version 0.1.7.

Fixed from v0.1.6

  • "Start with sample data" works on a Docker install. demo-warehouse now joins the connector network, so you no longer need the docker network connect workaround.

What changed

Batch pipelines

  • Reload onto object storage (GCS, S3, Azure Blob, MinIO): if the old files cannot all be deleted, the run fails. Before, it loaded on top of them.
  • Resume continues each table from where that table's own sweep stopped. A table whose row count is an exact multiple of the page size records that it finished.
  • Completion: a run completes only after its post-run row check passes; a failed check fails the run. The next run re-sends batches the destination refused in an earlier run.
  • MongoDB: a collection whose _id mixes types gets one column type per run.
  • Stop also stops the pipeline's sink worker.
  • Sink connector: it runs on every install, including Helm with connectors.cdc.enabled=false. A batch pipeline with a destination checks for it before it starts.
  • MinIO down: a run stops staging through it after the first failure, instead of retrying on every batch.
  • Removed: the STRICT_PREFLIGHT setting. It had no effect.

Change data capture

  • Behaviour change: keyless tables. A keyless table into a PostgreSQL, MySQL or MongoDB destination is now a warning, not a blocker. On such a table, an UPDATE adds a row and a DELETE is dead-lettered.
  • Declared keys only. The sink upserts only on a declared key, taken from the source's message key or the destination config. It never uses a guessed id column, which could merge distinct rows.
  • Keyless MongoDB rows are keyed on their Kafka record, so a replay does not duplicate them. MongoDB changes within the same second keep their order.
  • MySQL destination key columns are created indexable (VARCHAR), so an upsert matches its row instead of inserting it again on every replay.
  • Reload sends its re-snapshot only once the connector reports it is running.
  • Refused credentials: when a destination refuses the sink's credentials, the pipeline stops as needing configuration instead of retrying.
  • SQL Server: removing a table from a pipeline deletes its topic. The sink no longer creates phantom topics when it restarts after an edit.
  • CDC lag is reported in messages, from the newest reading.

Sink

  • Destination outage: a worker holding rows through an outage is no longer restarted as stalled.
  • One host per write: the first destination host that answers is final. A refused write is not sent again under another name for the same service. On Helm, the destination's real error is no longer replaced by no such host.
  • Layout v2 files are always Parquet. Before, a connection saved with format infer (the default in the GCS and Azure Blob forms) got gzip JSON under a .parquet name.

Connectors

  • Generate a connector from an OpenAPI document in the UI. Upload a file, paste it, or give a URL. Your browser fetches the URL, not the server.
  • Generated connectors deploy. Their files are saved readable by the orchestrator, and each is deployed from its saved version.
  • An upgrade runs the new connector code. The connector-seed image replaces each shipped connector version and keeps your generated connectors. A connector container is rebuilt when its code changes, not only when its version tag does.
  • In-cluster databases default to no TLS. A database inside the same Kubernetes cluster (*.svc, *.cluster.local) is treated like a Docker service name. An in-cluster MongoDB or PostgreSQL connects from host, port, user and password, without a connection string. An explicit TLS or sslmode setting still wins.

Interface

  • Pipeline pages say only what they measured.
    • A stage is marked done on its own evidence.
    • A CDC header reads "Setting up" during a Reload.
    • The header's row count and step agree with the table statistics and the timeline.
    • A status change shows within seconds.
  • Table statistics refresh while a run is going and take a final reading when it ends. CDC health reads the current run's probe.
  • Batch Checkpoints card: it stays readable with many tables, with a summary line, a filter and search, and one compact row per table.
  • Stop closes the actions menu. A CDC pipeline no longer sticks at "Failed" after Stop and Resume.
  • Chat finds a saved connection by its name, such as "Demo warehouse".
  • The /chat heading reads "Data Pipeline".

Self-hosting

  • Blob staging: blob passthrough stages through its own blob-staging MinIO, which connectors can reach. The platform MinIO stays off the connector network.
  • Start-up: the orchestrator waits for the database migrations before it runs its boot queries.
  • Alerts: a Postgres, Kafka or Kafka Connect that is still starting no longer raises a CRITICAL alert.
  • LLM cost logging works.
  • Installing from a checkout or a mirror: install.sh installs from a checkout (RSYNC_COMPOSE_DIR, with an explicit RSYNC_VERSION) or pulls images from a mirror (RSYNC_IMAGE_REGISTRY).

Kubernetes

  • Blob staging: the chart ships a dedicated blob-staging MinIO that only connector pods may reach under networkPolicy.enabled. install-k8s.sh generates its keys.
  • Generate is not available on Kubernetes yet. It refuses to save a connector and says why. Each pod keeps its own copy of the connector catalog, so a generated connector would not reach any other service.

Tested before release

Release candidates of this code were tested in three setups:

  • A fresh VM. Installed with install.sh from a checkout. We generated a connector from an OpenAPI document and deployed it, and it passed its connection test. The sample-data demo worked. Batch statistics covered all 42 tables. A PostgreSQL CDC insert, update and delete landed, and the keyless-table warning showed.
  • A v0.1.6 install upgraded in place. The upgrade replaced every shipped connector file with the new release's code. A connector container whose code had changed was rebuilt; on the next deploy, the unchanged image was reused.
  • A fresh GKE cluster with Google Managed Kafka (mTLS). We ran a Helm fresh install and a Helm upgrade. Also tested:
    • PostgreSQL → GCS CDC, including Reload;
    • MongoDB → GCS batch, which wrote Parquet (300 of 300 rows);
    • the Generate refusal;
    • the blob-staging network path under network policy.

The GKE run found the last two connector and sink fixes above: in-cluster databases default to no TLS, and layout v2 files are always Parquet. Both are covered by tests, but neither was re-run on a live cluster.

Assets 5
Loading