feat(cdc): connection pooling, tenant-scoped offsets, exact-once idempotency - #61
Merged
Conversation
…potency Completes the partial/open items from the architecture analysis: - F5/P2: restore end-to-end connection pooling. PooledJdbcBatchWriter now builds a Hikari pool from the actual ConnectionConfiguration (jdbcUrl + credentials) and keys it by destination identity, so separate containers get separate pools. PostgresWriter/MySqlWriter extend it; a Spring DisposableBean closes all pools on shutdown. Per-batch auto-commit: the SPI's commit()/rollback() are no-ops because each batch is one executeBatch() (ponytail: single-batch tx; multi-batch tx deferred). - D2/S3: scope the CDC offset store by tenant. Offsets are keyed by tenantId:pipelineId (73 chars) instead of pipelineId alone, so two tenants sharing a pipelineId no longer overwrite each other's resume position. cdc_offsets.pipeline_id widened to VARCHAR(128); CaptureEntry carries the tenant so global shutdown persists the scoped offset. - F14: wire the atomic exactly-once idempotency primitive. The post-success marking now uses markProcessedIfAbsent (INSERT ... ON CONFLICT DO NOTHING) keyed by eventId+pipeline bearing the tenant, so concurrent workers can't double-apply; retryEngine.success runs only for the marking call. Tests: CaptureLifecycle unit tests updated for scoped offset keys; full all-module integration suite green (F14/F5/D2 verified against real PG).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #60 (already merged). Completes the partial/open items from the architecture analysis that landed after #60 was merged.
Changes
PooledJdbcBatchWriterbuilds a Hikari pool from the actualConnectionConfiguration(jdbcUrl + credentials), keyed by destination identity so distinct destinations (e.g. separate test containers) get separate pools.PostgresWriter/MySqlWriterextend it; a SpringDisposableBeancloses all pools on shutdown. Per-batch auto-commit — each batch is a singleexecuteBatch(), so the SPIcommit()/rollback()are no-ops (ponytail:note added).tenantId:pipelineId(73 chars,cdc_offsets.pipeline_idwidened toVARCHAR(128)) instead of pipelineId alone, so two tenants sharing a pipelineId can't overwrite each other's resume position.CaptureEntrycarries the tenant so global shutdown persists the scoped key.markProcessedIfAbsent(INSERT ... ON CONFLICT DO NOTHING), previously dead code;retryEngine.successruns only for the marking call.Testing
Full all-module integration suite (
test -Dtests.integration=true) green on this branch (~560 tests). CaptureLifecycleUnitTest updated for scoped offset keys.