v1.4.0

⚡ Highlights

📊 Native Prometheus scrape endpoint
🎯 Granular object type filtering for pg-to-pg schema snapshots and DDL replication
🛡️ Transformation rules are now validated against unique indexes before a run starts, instead of failing mid-load
🔌 Target connection pool limits are configurable
🪝 Webhook delivery retries transient failures and no longer silently drops still-failing events; adds idempotency headers
🐛 Fixed a DDL parsing bug that silently dropped RENAME COLUMN / DROP COLUMN schema changes
🛂 New preflight check for CREATEROLE when a snapshot restores roles
Dependency updates

What's Changed

New features

• Add prometheus endpoint by @pauliyobo in #1008
• Granular object type filtering for pg to pg snapshot and streaming by @tsg in #735
• Add target CREATEROLE preflight check for roles snapshotting by @lghuy05 in #1055
• Make target connection limits configurable by @subotac in #1048
• Add a uniqueness classification to transformers by @kvch in #1082
• Classify the greenmask and neosync transformers for uniqueness by @kvch in #1084, #1085
• Validate transformation rules against unique indexes by @kvch in #1086

Bug fixes

• Reject masking configurations that mask no characters by @kvch in #1083
• Fix retry idempotency for webhook target by @kvch in #1080
• Stop the statement terminator being captured as part of a column name by @kvch in #1078

Documentation

• Document transformer uniqueness and the unique index check by @kvch in #1088
• Document the destructive --reset snapshot behaviour by @kvch in #1059

Dependency updates

• Bump actions/attest from 4.2.0 to 4.2.2 by @dependabot[bot] in #1065
• Bump go.opentelemetry.io/otel/sdk/metric from 1.44.0 to 1.45.0 by @dependabot[bot] in #1064
• Bump github.com/pgvector/pgvector-go/pgx from 0.4.0 to 0.4.1 by @dependabot[bot] in #1066
• Bump docker/login-action from 4.5.2 to 4.6.0 by @dependabot[bot] in #1061
• Bump clickhouse/clickhouse-server, opensearch and opensearch-dashboards build images by @dependabot[bot] in #1060, #1062, #1063

Details

Prometheus scrape endpoint

The setting instrumentation.metrics.prometheus.enabled exposes a /metrics endpoint on the existing health server. Metrics can now be scraped directly, with no collector in between:

instrumentation:
  metrics:
    prometheus:
      enabled: true # Defaults to false
      endpoint: "/metrics" # Defaults to /metrics

The endpoint returns 404 when disabled.

Object type filtering

Both the schema snapshot and DDL replication for pg-to-pg pipelines can now be restricted to a subset of object categories (tables, sequences, types, indexes, functions, views, triggers, etc.) via include_object_types (allowlist) or exclude_object_types (denylist), independently for snapshot and replication. A DDL statement that touches several object types (e.g. CREATE TABLE with a primary key) is only skipped when every object it touches is excluded.

Transformation rules are validated against unique indexes.

Anonymization is lossy by design: masking with type: id keeps a 6-character prefix, so every value sharing that prefix collapses to the same masked value. Applied to a column covered by a unique index, this previously surfaced as duplicate key value violates unique constraint hours into a data load. Every transformer is now classified as preserved (distinct inputs stay distinct), not_guaranteed (random or hashed output, collisions possible) or lossy (collides by construction).

validate rules and pipeline startup now read pg_index for the tables in scope and check the configured rules against unique indexes, primary keys and unique constraints. With a Postgres target, a lossy transformer on a covered column fails at parse time with transformation rules break a unique index; with other targets (Kafka, Elasticsearch/OpenSearch, webhooks) the same finding is a warning, since there is no unique index to violate. A not_guaranteed transformer always warns. Set allow_uniqueness_loss: true on a column rule to keep a lossy transformer anyway. A custom mask configured to mask zero characters (mask_begin equal to mask_end, or unmask_begin: 0) is now rejected at construction rather than silently passing values through unmasked. Use noop to pass a column through on purpose.

Configure target side connection count

target.postgres.max_connections (PGSTREAM_POSTGRES_WRITER_MAX_CONNECTIONS) now controls the writer pool size, defaulting to 50 and honoring pool_max_conns from the target URL when set explicitly. Previously this setting also sized the schema observer's pool, so a target capped at 90 connections to stay under a 100-connection server limit could still open up to 180. The schema observer is now sized separately, at min(writer pool, 16). So the process now opens at most the configured writer limit plus 16.

Webhook delivery

A failed webhook POST is now retried with configurable backoff (network errors, 429, 5xx); other non-2xx responses are treated as permanent and not retried. A delivery that is still failing after retries is logged and dropped so one broken subscriber can't block delivery to everyone else, and at-least-once semantics apply on restart. New X-Pgstream-LSN / Idempotency-Key headers let subscribers dedupe redelivered events (omitted for snapshot events, which share a zero LSN). DISABLE_RETRIES now actually disables retries.

New Contributors

• @pauliyobo made their first contribution in #1008
• @subotac made their first contribution in #1048

Full Changelog: v1.3.1...v1.4.0