Skip to content

fix: isolate ClickHouse publication#4014

Merged
vpol merged 3 commits into
mainfrom
codex/clickhouse-filtered-publication
Jul 21, 2026
Merged

fix: isolate ClickHouse publication#4014
vpol merged 3 commits into
mainfrom
codex/clickhouse-filtered-publication

Conversation

@vpol

@vpol vpol commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What changed

  • creates a dedicated all-tables clickhouse_sync publication
  • enables PostgreSQL 18 stored generated-column publishing
  • configures only ClickHouse sync to use the publication with connector autocreation disabled
  • retains table.include.list as the ClickHouse connector's table filter
  • replaces any legacy clickhouse_sync publication during migration
  • updates the shared CircleCI orb to 0.0.20, which pins the Google Cloud CLI version

Why

ClickHouse sync can be isolated from the shared dbz_publication without changing API Debezium. The publication must publish stored generated columns because post uses REPLICA IDENTITY FULL and contains generated slug; connector-created publications default to generated columns disabled and would block post updates.

Using an all-tables publication keeps the database migration independent of the connector's evolving table list. PostgreSQL can stream changes for every table to this replication slot, while Debezium emits only tables matched by the existing table.include.list.

The shared orb update avoids intermittent Docker and Pulumi failures caused by resolving the latest Google Cloud CLI version from its release-notes page.

Validation

  • complete 540-migration chain on PostgreSQL 18
  • migration rollback and reapply
  • legacy all-tables publication replacement
  • no-op post update succeeds with publish_generated_columns = stored
  • pnpm run build
  • pnpm run lint
  • CircleCI config validation with the private-orb organization context
  • Prettier and git diff --check

@pulumi

pulumi Bot commented Jul 21, 2026

Copy link
Copy Markdown

🍹 The Update (preview) for dailydotdev/api/prod (at fc4d8f3) was successful.

Resource Changes

    Name                                                       Type                           Operation
+   vpc-native-api-db-migration-2cc9e1bb                       kubernetes:batch/v1:Job        create
~   vpc-native-sync-subscription-with-cio-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-highlights-cron                         kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-digests-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-update-highlighted-views-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-ws-deployment                                   kubernetes:apps/v1:Deployment  update
~   vpc-native-private-deployment                              kubernetes:apps/v1:Deployment  update
~   vpc-native-update-current-streak-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-expire-super-agent-trial-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-clickhouse-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-rotate-weekly-quests-cron                       kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-history-clickhouse-cron  kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-user-companies-cron                kubernetes:batch/v1:CronJob    update
~   vpc-native-update-views-cron                               kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-old-notifications-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-update-source-public-threshold-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-interest-scheduled-run-cron                     kubernetes:batch/v1:CronJob    update
~   vpc-native-subscription-anniversary-achievements-cron      kubernetes:batch/v1:CronJob    update
+-  clickhouse-sync-clickhouse-sync-config                     kubernetes:core/v1:Secret      create-replacement
~   vpc-native-clean-zombie-opportunities-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-materialize-yearly-best-post-archives-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-generic-referral-reminder-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-achievements-cron                kubernetes:batch/v1:CronJob    update
~   vpc-native-user-posts-analytics-refresh-cron               kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                                      kubernetes:apps/v1:Deployment  update
~   vpc-native-rotate-daily-quests-cron                        kubernetes:batch/v1:CronJob    update
+-  clickhouse-sync-clickhouse-sync                            kubernetes:apps/v1:Deployment  create-replacement
~   vpc-native-clean-channel-highlights-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-check-analytics-report-cron                     kubernetes:batch/v1:CronJob    update
~   vpc-native-materialize-monthly-best-post-archives-cron     kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-clickhouse-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-generate-search-invites-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-validate-active-users-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-stale-user-transactions-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-update-tags-str-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-deployment                  kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-zombie-images-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-bg-deployment                                   kubernetes:apps/v1:Deployment  update
+   vpc-native-api-clickhouse-migration-2cc9e1bb               kubernetes:batch/v1:Job        create
~   vpc-native-squad-posts-analytics-refresh-cron              kubernetes:batch/v1:CronJob    update
... and 18 other changes

@vpol
vpol marked this pull request as ready for review July 21, 2026 08:35
@vpol
vpol force-pushed the codex/clickhouse-filtered-publication branch from d4a13f9 to 2423158 Compare July 21, 2026 08:53
@vpol
vpol merged commit 968cf39 into main Jul 21, 2026
9 checks passed
@vpol
vpol deleted the codex/clickhouse-filtered-publication branch July 21, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant