DBZ-7616 Add query timeout on JdbcConnection controlled by `query.timeout.ms` property #5483

mfvitale · 2024-04-12T08:06:33Z

closes: https://issues.redhat.com/browse/DBZ-7616

mfvitale · 2024-04-15T12:57:03Z

Note about the 'MariaDB/MySQL" changes:

Adding query timeout with statement.setQueryTimeout() will produce query prefixed with SET STATEMENT max_statement_time=60 FOR SELECT * for MariaDB.

Before this PR all statement starting with SET STATEMENT .* was skipped by ddl.filter properties due to some automatic query done by Amazon RDS prefixed with the query time out statement. The side effect of that was that an user DDL starting with SET STATEMENT .* was also, IMHO, wrongly skipped.

After this PR just the statement from Amazon RDS will be skipped meanwhile the others will be correctly processed.

@Naros This can maybe be relevant for your Maria DB stuff.

...nnector-mysql/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java

mfvitale · 2024-04-15T14:14:27Z

Seems that this change lead to an issue with Informix

After sometimes I started to see tests failing with

2024-04-15 16:10:07,205 ERROR IncrementalSnapshotIT||engine Engine has failed with  [io.debezium.embedded.async.AsyncEmbeddedEngine]
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Couldn't obtain database name
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.startSourceTasks(AsyncEmbeddedEngine.java:411)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:202)
	at io.debezium.embedded.async.TestingAsyncEmbeddedEngine.run(TestingAsyncEmbeddedEngine.java:27)
	at io.debezium.embedded.AbstractConnectorTest.lambda$start$8(AbstractConnectorTest.java:423)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Couldn't obtain database name
	at io.debezium.connector.informix.InformixConnection.retrieveRealDatabaseName(InformixConnection.java:112)
	at io.debezium.connector.informix.InformixConnection.<init>(InformixConnection.java:80)
	at io.debezium.connector.informix.InformixConnectorTask.lambda$start$0(InformixConnectorTask.java:73)
	at io.debezium.jdbc.DefaultMainConnectionProvidingConnectionFactory.newConnection(DefaultMainConnectionProvidingConnectionFactory.java:26)
	at io.debezium.connector.informix.InformixConnectorTask.start(InformixConnectorTask.java:97)
	at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:240)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.lambda$startSourceTasks$2(AsyncEmbeddedEngine.java:394)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 3 common frames omitted
Caused by: java.sql.SQLException: Cannot create a user thread.
	at com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1348)
	at com.informix.jdbc.IfxDriver.connect(IfxDriver.java:167)
	at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:246)
	at io.debezium.jdbc.JdbcConnection$ConnectionFactoryDecorator.connect(JdbcConnection.java:132)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:894)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:889)
	at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:633)
	at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:506)
	at io.debezium.connector.informix.InformixConnection.retrieveRealDatabaseName(InformixConnection.java:109)
	... 12 common frames omitted
Caused by: com.informix.asf.IfxASFRemoteException: 
	at com.informix.asf.Connection.recvConnectionResponse(Connection.java:589)
	at com.informix.asf.Connection.establishConnection(Connection.java:1418)
	at com.informix.asf.Connection.connect(Connection.java:333)
	at com.informix.asf.Connection.<init>(Connection.java:325)
	at com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1248)
	... 20 common frames omitted

and the database starts to log

14:09:49.083 IFX >>>2024-04-15 14:09:49.083  Warning: The IBM IDS Developer Edition license restriction limits
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  the number of connections to this server to 25.
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  You already have 25 connections at this time. Any further connections
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  attempts will be rejected.

@nrkljo any hint/idea?

Naros · 2024-04-15T17:10:21Z

Adding a note here so it gets recorded in the PR #5434, need to review these and incorporate once merged.

mfvitale · 2024-04-16T09:44:00Z

Seems that this change lead to an issue with Informix

After sometimes I started to see tests failing with

2024-04-15 16:10:07,205 ERROR IncrementalSnapshotIT||engine Engine has failed with  [io.debezium.embedded.async.AsyncEmbeddedEngine]
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Couldn't obtain database name
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.startSourceTasks(AsyncEmbeddedEngine.java:411)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:202)
	at io.debezium.embedded.async.TestingAsyncEmbeddedEngine.run(TestingAsyncEmbeddedEngine.java:27)
	at io.debezium.embedded.AbstractConnectorTest.lambda$start$8(AbstractConnectorTest.java:423)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Couldn't obtain database name
	at io.debezium.connector.informix.InformixConnection.retrieveRealDatabaseName(InformixConnection.java:112)
	at io.debezium.connector.informix.InformixConnection.<init>(InformixConnection.java:80)
	at io.debezium.connector.informix.InformixConnectorTask.lambda$start$0(InformixConnectorTask.java:73)
	at io.debezium.jdbc.DefaultMainConnectionProvidingConnectionFactory.newConnection(DefaultMainConnectionProvidingConnectionFactory.java:26)
	at io.debezium.connector.informix.InformixConnectorTask.start(InformixConnectorTask.java:97)
	at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:240)
	at io.debezium.embedded.async.AsyncEmbeddedEngine.lambda$startSourceTasks$2(AsyncEmbeddedEngine.java:394)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 3 common frames omitted
Caused by: java.sql.SQLException: Cannot create a user thread.
	at com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1348)
	at com.informix.jdbc.IfxDriver.connect(IfxDriver.java:167)
	at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:246)
	at io.debezium.jdbc.JdbcConnection$ConnectionFactoryDecorator.connect(JdbcConnection.java:132)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:894)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:889)
	at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:633)
	at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:506)
	at io.debezium.connector.informix.InformixConnection.retrieveRealDatabaseName(InformixConnection.java:109)
	... 12 common frames omitted
Caused by: com.informix.asf.IfxASFRemoteException: 
	at com.informix.asf.Connection.recvConnectionResponse(Connection.java:589)
	at com.informix.asf.Connection.establishConnection(Connection.java:1418)
	at com.informix.asf.Connection.connect(Connection.java:333)
	at com.informix.asf.Connection.<init>(Connection.java:325)
	at com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1248)
	... 20 common frames omitted

and the database starts to log

14:09:49.083 IFX >>>2024-04-15 14:09:49.083  Warning: The IBM IDS Developer Edition license restriction limits
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  the number of connections to this server to 25.
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  You already have 25 connections at this time. Any further connections
14:09:49.083 IFX >>>2024-04-15 14:09:49.083  attempts will be rejected.

@nrkljo any hint/idea?

This should solve the Informix issue debezium/debezium-connector-informix#46

mfvitale · 2024-04-16T09:44:41Z

/packit test --labels oracle

nrkljo · 2024-04-16T13:11:17Z

@mfvitale I believe you're on to something there. The connection limit in Informix Developer Edition sometimes creates problems and better reuse of connections should adress this, to som extent...

mfvitale · 2024-04-16T14:26:33Z

/packit test --labels oracle

mfvitale · 2024-04-17T07:09:06Z

/packit test --identifier oracle-21-xs

mfvitale · 2024-04-17T13:59:36Z

/packit test --labels oracle

…eout.ms` property

… just the ones related to Amazon RDS

mfvitale · 2024-04-18T11:38:40Z

/packit test --labels oracle

jpechane

@mfvitale LGTM, thanks! I left one comment related to using inhertiance instead of condition. When addressed and CI passes please feel free to merge PR yourself.

debezium-connector-binlog/src/test/java/io/debezium/connector/binlog/BinlogConnectorIT.java

mfvitale · 2024-04-19T06:56:25Z

/packit test --labels oracle

mfvitale force-pushed the DBZ-7616 branch 2 times, most recently from 7421997 to e2ec02a Compare April 12, 2024 08:08

mfvitale marked this pull request as draft April 12, 2024 08:12

mfvitale force-pushed the DBZ-7616 branch 3 times, most recently from fc6566e to a2cdefb Compare April 15, 2024 10:33

mfvitale commented Apr 15, 2024

View reviewed changes

...nnector-mysql/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java Outdated Show resolved Hide resolved

mfvitale mentioned this pull request Apr 16, 2024

DBZ-7616 Add query timeout property debezium/debezium-connector-informix#46

Merged

mfvitale marked this pull request as ready for review April 17, 2024 13:45

mfvitale force-pushed the DBZ-7616 branch from 870b183 to 1b70027 Compare April 17, 2024 13:59

mfvitale added 3 commits April 18, 2024 11:49

DBZ-7616 Add query timeout on JdbcConnection controlled by `query.tim…

2de8c8a

…eout.ms` property

DBZ-7616 Avoid to filter out every SET STATEMENT queries and filter…

31837ff

… just the ones related to Amazon RDS

DBZ-7616 Simplify Oracle tests assertion on SQLTimeoutException

9183855

mfvitale force-pushed the DBZ-7616 branch from 1b70027 to b18542c Compare April 18, 2024 10:39

jpechane reviewed Apr 18, 2024

View reviewed changes

debezium-connector-binlog/src/test/java/io/debezium/connector/binlog/BinlogConnectorIT.java Outdated Show resolved Hide resolved

DBZ-7616 Align query timeout changes to MariaDB connector

828960f

mfvitale force-pushed the DBZ-7616 branch from b18542c to 828960f Compare April 18, 2024 13:57

mfvitale mentioned this pull request Apr 19, 2024

DBZ-7616 Document database.query.timeout.ms property #5498

Merged

mfvitale merged commit 7ee8935 into debezium:main Apr 22, 2024
36 of 39 checks passed

mfvitale deleted the DBZ-7616 branch April 22, 2024 07:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBZ-7616 Add query timeout on JdbcConnection controlled by `query.timeout.ms` property #5483

DBZ-7616 Add query timeout on JdbcConnection controlled by `query.timeout.ms` property #5483

mfvitale commented Apr 12, 2024

mfvitale commented Apr 15, 2024 •

edited

mfvitale commented Apr 15, 2024

Naros commented Apr 15, 2024

mfvitale commented Apr 16, 2024

mfvitale commented Apr 16, 2024

nrkljo commented Apr 16, 2024

mfvitale commented Apr 16, 2024

mfvitale commented Apr 17, 2024

mfvitale commented Apr 17, 2024

mfvitale commented Apr 18, 2024

jpechane left a comment

mfvitale commented Apr 19, 2024

DBZ-7616 Add query timeout on JdbcConnection controlled by query.timeout.ms property #5483

DBZ-7616 Add query timeout on JdbcConnection controlled by query.timeout.ms property #5483

Conversation

mfvitale commented Apr 12, 2024

mfvitale commented Apr 15, 2024 • edited

mfvitale commented Apr 15, 2024

Naros commented Apr 15, 2024

mfvitale commented Apr 16, 2024

mfvitale commented Apr 16, 2024

nrkljo commented Apr 16, 2024

mfvitale commented Apr 16, 2024

mfvitale commented Apr 17, 2024

mfvitale commented Apr 17, 2024

mfvitale commented Apr 18, 2024

jpechane left a comment

Choose a reason for hiding this comment

mfvitale commented Apr 19, 2024

DBZ-7616 Add query timeout on JdbcConnection controlled by `query.timeout.ms` property #5483

DBZ-7616 Add query timeout on JdbcConnection controlled by `query.timeout.ms` property #5483

mfvitale commented Apr 15, 2024 •

edited