[Source-postgres]: Add logging in case of multiple records with same LSN #35842
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
octavia-squidington-iii
added
area/connectors
| @@ -57,6 +57,7 @@ private ContainerModifier(String methodName) { | |||
|
|
|||
| } | |||
|
|
|||
| @SuppressWarnings("deprecation") | |||
There was a problem hiding this comment.
I'm not quite sure, but this underlying library seems to have a deprecation that is failing the gradle check, so it i's necessary
octavia-squidington-iii
added
the
area/documentation
|
/publish-java-cdk
|
| if (targetPosition.isEventAheadOffset(checkpointOffsetToSend, event)) { | ||
| sendCheckpointMessage = true; | ||
| } else { | ||
| LOGGER.info("Encountered {} records with the same event offset", recordsLastSync); |
There was a problem hiding this comment.
This log message is causing our airbyte logs to significantly grow in size as we commonly see this message >1 million times in each sync.
Is it possible to selectively disable this log or would we have to change our log level? We don't believe multiple records with the same offset is an issue for us because most of our database operations are batch inserts.
There was a problem hiding this comment.
@WillKoehrsen are you able to submit a PR which limits how often this is logged? Maybe only once per offset or stop after 100 logs?
There was a problem hiding this comment.
Sorry, I don't have the experience with the codebase to feel comfortable opening a PR. We were able to get around the issue in the short term by downgrading our postgres connector to 3.3.12.
There was a problem hiding this comment.
ok, we'll track this bug via #36903
There was a problem hiding this comment.
@WillKoehrsen we have removed this log line in the new version of source-postgres
There was a problem hiding this comment.
Thanks a lot! Really appreciate it.
| testExecutionConcurrency=-1 | ||
|
|
||
| JunitMethodExecutionTimeout=5 m |
There was a problem hiding this comment.
@akashkulk what's this for?
There was a problem hiding this comment.
In this PR, CDK 0.23.15 #35827 default timeout was reduced from 5m to 1m by default. It wasn't really validated against any other certified connectors other than source-mssql for which it seems like this fix exclusively applied.
This timeout just adds back that functionality.
There was a problem hiding this comment.
No description provided.