v0.1.3
Thirteen changes since v0.1.2, centred on making MongoDB → GCS CDC correct for
multi-collection pipelines, plus operator notifications.
Data correctness (CDC)
- #48 — a locked
destination_namespacewas silently discarded on three
destinations. A pipeline that declared and locked a destination namespace
had its CDC rows written into the connection's database/dataset/schema
instead. Nothing downstream can detect that: the rows land, the counts match,
the lag is zero, and they are in the wrong place. Two independent halves, both
fixed — the sink's document-DB CDC branch never forwarded the namespace (while
its own batch write and reload-drop always did), and mongodb / bigquery /
redshift accepted the parameter and ignored it. The other ten destinations were
censused; object stores stay deliberately silent because they key by the source
table id. Both write lanes — CDC and batch — are now guarded by mutation-proven
tests. - #42 — the sink derived its topic prefix by hand and subscribed to topics
nobody writes. Self-masking, because the pre-create step then created the
bogus topics: the pipeline reported Running / Healthy / 100% while delivering
zero rows. - #45 — every bronze object was written uncompressed, and a compressed parquet
was unreadable. Billed twice over (object storage plus bytes scanned) and the
compressed path did not round-trip. - #39 — MongoDB → GCS CDC fixes and failure notifications.
Notifications
- #43 — admin-configured Slack and SMTP delivery, with email controls, so a
stopped or failing pipeline reaches the team instead of sitting silent.
Deployment and self-host
- #47 — publish arm64 images, fail fast on an unreachable database, bind dev
ports to loopback. - #41 — the one container that resolves connectors was the one with no
connector mount, which made no pipeline creatable on a quickstart install. - #40 — arm64 platform pin and MinIO environment-variable fixes.
Explorer, docs and dependencies
- #46 document browse mode for MongoDB connections · #44 README leads with
"AI-native data pipeline platform" - #33, #50, #51 — dependency bumps (actions, pip, npm).
Upgrading
Image tags move to v0.1.3. The sink worker and the mongodb / bigquery / redshift
connector images must be pulled for the namespace fix to take effect — a pipeline
that sets no destination namespace is unaffected by it.