fix: Disable CDC checkpointing using a flag - #24275
Conversation
…ng Offset file during syncs
Octavia Squidington III (octavia-squidington-iii)
added
area/connectors
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-alloydb |
2.0.3 |
✅ | ✅ |
source-alloydb-strict-encrypt |
2.0.3 |
🔵 (ignored) |
🔵 (ignored) |
source-mssql |
1.0.4 |
✅ | ✅ |
source-mssql-strict-encrypt |
1.0.4 |
🔵 (ignored) |
🔵 (ignored) |
source-mysql |
2.0.5 |
✅ | ✅ |
source-mysql-strict-encrypt |
2.0.5 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres |
2.0.4 |
✅ | ✅ |
source-postgres-strict-encrypt |
2.0.4 |
🔵 (ignored) |
🔵 (ignored) |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
|
/test connector=connectors/source-postgres-strict-encrypt
Build PassedTest summary info: |
|
/test connector=connectors/source-mysql-strict-encrypt
Build PassedTest summary info: |
|
/test connector=connectors/source-mssql-strict-encrypt
Build PassedTest summary info: |
|
/test connector=connectors/source-alloydb
Build PassedTest summary info: |
|
/test connector=connectors/source-alloydb-strict-encrypt
Build PassedTest summary info: |
Octavia Squidington III (octavia-squidington-iii)
added
area/documentation
|
/publish connector=connectors/source-alloydb
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mysql
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mssql
|
|
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-alloydb-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mysql-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
Definitions generated manually |
Generated definitions manually |
Published fine, Generated definitions manually |
|
/publish connector=connectors/source-mssql-strict-encrypt auto-bump-version=false run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false run-tests=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
* Disable CDC checkpointing until finished the investigation with reading Offset file during syncs * Bump connector versions * auto-bump connector version * Publish postgres manually * Definitions manually for mysql * Generated definitions manually mssql --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Disable CDC checkpointing until finished the investigation with reading Offset file during syncs * Bump connector versions * auto-bump connector version * Publish postgres manually * Definitions manually for mysql * Generated definitions manually mssql --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
3 participants
What
There is an issue when we try to read the offset file at the same time Debezium is writing into it, making syncs to fail.
How
We disable CDC checkpointing using a flag until we finished the investigation with reading Offset file during syncs.
Also, with this solution, it can be enabled per each connector individually, so only one connector will be impacted at the time.
Test
Performed the following tests:
See image below:
