Skip to content

v0.1.3

Choose a tag to compare

@rahulv8 rahulv8 released this 15 Sep 21:18
· 0 commits to 6e831623705c046d88b7f89943fcf4ebede77215 since this release
00220cc

Thirteen changes since v0.1.2, centred on making MongoDB → GCS CDC correct for
multi-collection pipelines, plus operator notifications.

Data correctness (CDC)

  • #48 — a locked destination_namespace was silently discarded on three
    destinations.
    A pipeline that declared and locked a destination namespace
    had its CDC rows written into the connection's database/dataset/schema
    instead. Nothing downstream can detect that: the rows land, the counts match,
    the lag is zero, and they are in the wrong place. Two independent halves, both
    fixed — the sink's document-DB CDC branch never forwarded the namespace (while
    its own batch write and reload-drop always did), and mongodb / bigquery /
    redshift accepted the parameter and ignored it. The other ten destinations were
    censused; object stores stay deliberately silent because they key by the source
    table id. Both write lanes — CDC and batch — are now guarded by mutation-proven
    tests.
  • #42 — the sink derived its topic prefix by hand and subscribed to topics
    nobody writes.
    Self-masking, because the pre-create step then created the
    bogus topics: the pipeline reported Running / Healthy / 100% while delivering
    zero rows.
  • #45 — every bronze object was written uncompressed, and a compressed parquet
    was unreadable.
    Billed twice over (object storage plus bytes scanned) and the
    compressed path did not round-trip.
  • #39 — MongoDB → GCS CDC fixes and failure notifications.

Notifications

  • #43 — admin-configured Slack and SMTP delivery, with email controls, so a
    stopped or failing pipeline reaches the team instead of sitting silent.

Deployment and self-host

  • #47 — publish arm64 images, fail fast on an unreachable database, bind dev
    ports to loopback.
  • #41 — the one container that resolves connectors was the one with no
    connector mount
    , which made no pipeline creatable on a quickstart install.
  • #40 — arm64 platform pin and MinIO environment-variable fixes.

Explorer, docs and dependencies

  • #46 document browse mode for MongoDB connections · #44 README leads with
    "AI-native data pipeline platform"
  • #33, #50, #51 — dependency bumps (actions, pip, npm).

Upgrading

Image tags move to v0.1.3. The sink worker and the mongodb / bigquery / redshift
connector images must be pulled for the namespace fix to take effect — a pipeline
that sets no destination namespace is unaffected by it.