[CDCSDK] CDC nemesis case fails with org.apache.kafka.connect.errors.ConnectException: Unable to obtain valid replication slot.

[shamanthchandra-yb]

Jira Link: DB-10655

Description

Please find stress link in JIRA description.

There was nemesis happened, and we observed below error during that time:

ERROR: Shutdown connection

However, after that, we saw sequence of logs stating:

2024-04-01 18:33:12,311 INFO   Postgres|ybconnector_cdc_1822a3_test_cdc_abc95e|postgres-connector-task  Unable to find confirmed_flushed_lsn, falling back to restart_lsn   [io.debezium.connector.postgresql.connection.PostgresConnection]
2024-04-01 18:33:12,311 WARN   Postgres|ybconnector_cdc_1822a3_test_cdc_abc95e|postgres-connector-task  Cannot obtain valid replication slot 'rs_cdc_1822a3_7be8_from_con' for plugin 'pgoutput' and database 'cdc_1822a3' [during attempt 90 out of 90, concurrent tx probably blocks taking snapshot.   [io.debezium.connector.postgresql.connection.PostgresConnection]
2024-04-01 18:33:12,367 INFO   Postgres|ybconnector_cdc_1822a3_test_cdc_f56770|postgres-connector-task  Failed to obtain valid replication slot, confirmed flush lsn is null   [io.debezium.connector.postgresql.connection.PostgresConnection]

and finally failing with

2024-04-01 18:33:14,068 ERROR  ||  WorkerSourceTask{id=ybconnector_cdc_1822a3_test_cdc_9b79d0-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
org.apache.kafka.connect.errors.ConnectException: Unable to obtain valid replication slot. Make sure there are no long-running transactions running in parallel as they may hinder the allocation of the replication slot when starting this connector
        at io.debezium.connector.postgresql.connection.PostgresConnection.readReplicationSlotInfo(PostgresConnection.java:358)
        at io.debezium.connector.postgresql.connection.PostgresConnection.getReplicationSlotState(PostgresConnection.java:281)
        at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:136)
        at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:268)
        at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:178)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:469)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:357)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)
        at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Source connector version

fourpointfour/ybdb-debezium:0.2

Connector configuration

add connector connector_name='ybconnector_cdc_1822a3_test_cdc_9b79d0' stream_id='rs_cdc_1822a3_e22a' db_name='cdc_1822a3' connector_host='172.151.20.132' table_list=['test_cdc_9b79d0'] {'name': 'ybconnector_cdc_1822a3_test_cdc_9b79d0', 'config': {'database.master.addresses': '172.151.29.64:7100,172.151.25.56:7100,172.151.24.69:7100', 'database.hostname': '172.151.29.64:5433,172.151.25.56:5433,172.151.24.69:5433', 'database.port': 5433, 'database.masterhost': '172.151.25.56', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_1822a3', 'snapshot.mode': 'initial', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '5', 'connector.class': 'io.debezium.connector.postgresql.PostgresConnector', 'topic.prefix': 'ybconnector_cdc_1822a3_test_cdc_9b79d0', 'plugin.name': 'pgoutput', 'slot.name': 'rs_cdc_1822a3_e22a_from_con', 'publication.autocreate.mode': 'filtered', 'publication.name': 'pn_ybconnector_cdc_1822a3_test_cdc_9b79d0', 'table.include.list': 'public.test_cdc_9b79d0'}}

YugabyteDB version

2.23.0.0-b86

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.