[CDCSDK] Regression (2.20.2): Consistency connector failed with 'Merger tried to set tablet safetime to a lower value'

[shamanthchandra-yb]

Jira Link: DB-9342

Description

Please find JIRA for run link.

ERROR  YugabyteDB|db_cdc|streaming|0  Producer failure   [io.debezium.pipeline.ErrorHandler]
io.debezium.DebeziumException: java.lang.AssertionError: Merger tried to set tablet safetime to a lower value. Tablet: 4ce25ac5128842a291019610133b9f8a Current safetime value: 6975384931147321344 Attempted set value: 6975384903738490880
	at io.debezium.connector.yugabytedb.YugabyteDBConsistentStreamingSource.getChanges2(YugabyteDBConsistentStreamingSource.java:282)
	at io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource.execute(YugabyteDBStreamingChangeEventSource.java:137)
	at io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource.execute(YugabyteDBStreamingChangeEventSource.java:47)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
	at io.debezium.connector.yugabytedb.YugabyteDBChangeEventSourceCoordinator.executeChangeEventSources(YugabyteDBChangeEventSourceCoordinator.java:150)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: Merger tried to set tablet safetime to a lower value. Tablet: 4ce25ac5128842a291019610133b9f8a Current safetime value: 6975384931147321344 Attempted set value: 6975384903738490880
	at io.debezium.connector.yugabytedb.consistent.Merger.setTabletSafeTime(Merger.java:80)
	at io.debezium.connector.yugabytedb.consistent.Merger.addMessage(Merger.java:43)
	at io.debezium.connector.yugabytedb.YugabyteDBConsistentStreamingSource.getChanges2(YugabyteDBConsistentStreamingSource.java:231)
	... 10 more

There is a closed GH #17414, but I am not sure if this is also because of large transactions.

Source connector version

1.9.5.y.34-SNAPSHOT

Connector configuration

add connector connector_name='yugabyte_connector' stream_id='415f33af7567461ca90e432a94d5fa14' db_name='cdc_bc160f' connector_host='172.151.19.59' table_list=['test_cdc_418388'] {'name': 'yugabyte_connector', 'config': {'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBConnector', 'database.hostname': '172.151.27.126:5433,172.151.16.112:5433,172.151.18.5:5433', 'database.master.addresses': '172.151.27.126:7100,172.151.16.112:7100,172.151.18.5:7100', 'database.port': 5433, 'database.masterhost': '172.151.16.112', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_bc160f', 'database.server.name': 'db_cdc', 'database.streamid': '415f33af7567461ca90e432a94d5fa14', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 1, 'topic.creation.default.replication.factor': '1', 'tasks.max': '1', 'table.include.list': 'public.test_cdc_418388', 'transforms': 'Reroute', 'transforms.Reroute.topic.regex': '(.*)', 'transforms.Reroute.topic.replacement': 'db_cdc_all_events', 'transforms.Reroute.type': 'io.debezium.transforms.ByLogicalTableRouter', 'transforms.Reroute.key.field.regex': 'db_cdc(.*)', 'transforms.Reroute.key.field.replacement': '$1', 'transaction.ordering': 'true', 'provide.transaction.metadata': 'true', 'max.poll.interval.ms': '5000'}}

YugabyteDB version

2.20.2.0-b43

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.