add configuration for cdc queue size #35739
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
octavia-squidington-iii
added
area/connectors
| import java.util.HashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.*; |
There was a problem hiding this comment.
Is this a formatting error?
There was a problem hiding this comment.
I checked as well.
the formatter actually is the one that put it in place.
I think we either reached some threshold
There was a problem hiding this comment.
that's super annoying if the formatter adds files to your PR...
| @@ -167,7 +163,7 @@ public static List<AutoCloseableIterator<AirbyteMessage>> getCdcReadIterators(fi | |||
| true, | |||
| firstRecordWaitTime, | |||
| subsequentRecordWaitTime, | |||
| AirbyteDebeziumHandler.QUEUE_CAPACITY, | |||
| queueSize, | |||
There was a problem hiding this comment.
I'll just use this opportunity to ask questions:
From my understanding, this underlying queue is a BlockingQueue only used outside of debezium - in other words we do not change debezium properties like max.queue.size right?
And since this basically defines the size of the queue, it could mean -
- the larger the number is, we could probably benefit from having fewer bottleneck issues
- the lower the number is, the less preserved memory it could cost.
Is that right? In that case, I feel we could benefit from gathering data on the max queue size used in the sync
There was a problem hiding this comment.
You're right in what you're saying. The queue we control is our queue buffering events coming out of debezium.
The reason I suspect this is the queue that causes an OOM is that the size of max.queue.size is a constant of 8K and we're also setting the max.queue.size.in.bytes which sets another limit of size (the first max reached is the limit).
|
let's add a counter in the queue implementation and put some data in the logs, so that we don't have to keep guessing in the future. We should be able to easily gather number of elements in the queue and total size in the queue |
|
/publish-java-cdk
|
stephane-airbyte
force-pushed
the
21498-source-mssql-cdc-job-is-getting-failed-due-to-oom-error
branch
from
5be001c
to
2fa69df
Compare
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @rodireich and the rest of your teammates on |
stephane-airbyte
force-pushed
the
21498-source-mssql-cdc-job-is-getting-failed-due-to-oom-error
branch
2 times, most recently
from
dfc3860
to
74fda84
Compare
|
/publish-java-cdk
|
stephane-airbyte
force-pushed
the
21498-source-mssql-cdc-job-is-getting-failed-due-to-oom-error
branch
from
74fda84
to
a48a677
Compare
|
/publish-java-cdk force=true
|
| @@ -127,8 +127,7 @@ public final C shared(String imageName, List<? extends NamedContainerModifier<C> | |||
| @SuppressWarnings("unchecked") | |||
| @Deprecated | |||
| public final C exclusive(String imageName, String... methods) { | |||
| return exclusive(imageName, | |||
| (NamedContainerModifier<C>) Stream.of(methods).map(n -> new NamedContainerModifierImpl<C>(n, resolveModifierByName(n))).toList()); | |||
| return exclusive(imageName, Stream.of(methods).map(n -> new NamedContainerModifierImpl<C>(n, resolveModifierByName(n))).toList()); | |||
There was a problem hiding this comment.
Thanks this was failing on warning
|
/approve-and-merge reason="source-mssql tests are still broken" |
|
/approve-and-merge reason="source-mssql tests are still broken" |
|
Lets do it! |
octavia-approvington
deleted the
21498-source-mssql-cdc-job-is-getting-failed-due-to-oom-error
branch
What
Similar to other connectors we allow an admin to configure mssql source's queue size used to buffer cdc events to help in case of OOM.
How
Add queue size configuration used by debezium handler rather than hard coding the size to 10k