fix debezium reader thread lake #1358

Matrix42 · 2022-07-10T15:59:09Z

this fix #1350

ruanhang1993

Thanks for your work, @Matrix42 . I have left some comments.

ruanhang1993 · 2022-08-01T02:21:33Z

...-cdc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java

+            // while loop in MySqlStreamingChangeEventSource's execute method
+            currentTaskRunning = false;
+            executor.shutdown();
+            executor.awaitTermination(5, TimeUnit.SECONDS);


This timeout should be saved as a constant.

ruanhang1993 · 2022-08-01T02:22:43Z

...-cdc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java

+            // while loop in MySqlStreamingChangeEventSource's execute method
+            currentTaskRunning = false;
+            executor.shutdown();
+            executor.awaitTermination(5, TimeUnit.SECONDS);


Maybe we could add a log when the awaitTermination method failed.

ruanhang1993 · 2022-08-01T02:24:55Z

...dc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/SnapshotSplitReader.java

@@ -330,6 +331,8 @@ public void close() {
            if (statefulTaskContext.getBinaryLogClient() != null) {
                statefulTaskContext.getBinaryLogClient().disconnect();
            }
+            executor.shutdown();
+            executor.awaitTermination(5, TimeUnit.SECONDS);


Same commnets in the BinlogSplitReader.java.

.../src/test/java/com/ververica/cdc/connectors/mysql/debezium/reader/BinlogSplitReaderTest.java

Matrix42 · 2022-08-01T13:55:42Z

Hi @ruanhang1993! Thanks for your review. I'v addressed the issue you mentioned, I'll rebase the master when the compile problem resolved.

Matrix42 · 2022-08-02T07:03:57Z

Hi @ruanhang1993! Please take a look when you are available. The test failed is same as #1144

CC @leonardBang

leonardBang · 2022-08-02T07:14:43Z

...dc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/SnapshotSplitReader.java

+            executor.shutdown();
+            boolean isShutdown =
+                    executor.awaitTermination(
+                            EXECUTOR_AWAIT_TERMINATION_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+            if (!isShutdown) {
+                LOG.warn("The thread executor of SnapshotSplitReader wasn't shutdown properly.");
+            }


Please check not NULL before call it.

Suggested change

executor.shutdown();

boolean isShutdown =

executor.awaitTermination(

EXECUTOR_AWAIT_TERMINATION_TIMEOUT_SECONDS, TimeUnit.SECONDS);

if (!isShutdown) {

LOG.warn("The thread executor of SnapshotSplitReader wasn't shutdown properly.");

}

if (executorService != null) {

executorService.shutdown();

if (!executorService.awaitTermination(

READER_CLOSE_TIMEOUT, TimeUnit.MILLISECONDS)) {

LOG.warn(

"Failed to close the snapshot split reader in {} ms.",

READER_CLOSE_TIMEOUT);

}

}

leonardBang · 2022-08-02T07:17:10Z

...dc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/SnapshotSplitReader.java

@@ -86,6 +87,8 @@ public class SnapshotSplitReader implements DebeziumReader<SourceRecord, MySqlSp
    public AtomicBoolean hasNextElement;
    public AtomicBoolean reachEnd;

+    private static final long EXECUTOR_AWAIT_TERMINATION_TIMEOUT_SECONDS = 5;


Suggested change

private static final long EXECUTOR_AWAIT_TERMINATION_TIMEOUT_SECONDS = 5;

private static final long READER_CLOSE_TIMEOUT = 30L;

And let's rename executor to executorService for more precise meaning.

leonardBang · 2022-08-02T07:24:20Z

...r-mysql-cdc/src/test/java/com/ververica/cdc/connectors/mysql/source/MySqlSourceTestBase.java

@@ -86,4 +89,11 @@ public static void assertEqualsInOrder(List<String> expected, List<String> actua
        assertEquals(expected.size(), actual.size());
        assertArrayEquals(expected.toArray(new String[0]), actual.toArray(new String[0]));
    }
+
+    public static void assertExecutorIsTerminated(DebeziumReader<?, ?> reader) throws Exception {
+        Field executorField = reader.getClass().getDeclaredField("executor");


Use reflection is not cool here, if you really want to check executorService member of SnapshotReader, you can expose a method in SnapshotReader like:

@VisibleForTesting public ExecutorService getExecutorService() { return executorService; }

leonardBang · 2022-08-02T07:57:12Z

...-cdc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java

+                executorService.shutdown();
+                if (executorService.awaitTermination(READER_CLOSE_TIMEOUT, TimeUnit.SECONDS)) {
+                    LOG.warn(
+                            "Failed to close the binlog split reader in {} ms.",


Suggested change

"Failed to close the binlog split reader in {} ms.",

"Failed to close the binlog split reader in {} seconds.",

leonardBang · 2022-08-02T07:57:41Z

...dc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/reader/SnapshotSplitReader.java

+                executorService.shutdown();
+                if (executorService.awaitTermination(READER_CLOSE_TIMEOUT, TimeUnit.SECONDS)) {
+                    LOG.warn(
+                            "Failed to close the snapshot split reader in {} ms.",


Suggested change

"Failed to close the snapshot split reader in {} ms.",

"Failed to close the snapshot split reader in {} seconds.",

leonardBang · 2022-08-02T07:59:41Z

...rc/test/java/com/ververica/cdc/connectors/mysql/debezium/reader/SnapshotSplitReaderTest.java

+        assertNull(snapshotSplitReader.getExecutorService());
+        assertTrue(snapshotSplitReader.getExecutorService().isTerminated());


? assertNotNull

leonardBang · 2022-08-02T07:59:49Z

.../src/test/java/com/ververica/cdc/connectors/mysql/debezium/reader/BinlogSplitReaderTest.java

+        assertNull(snapshotSplitReader.getExecutorService());
+        assertTrue(snapshotSplitReader.getExecutorService().isTerminated());


? assertNotNull

yes, I uesd wrong method.

leonardBang

Thanks @Matrix42 for the timely update, LGTM, let's wait the CI pass.

Matrix42 · 2022-08-02T08:05:30Z

Thanks @leonardBang for the review!

Matrix42 changed the title ~~fix SnapshotSplitReader and BinlogSplitReader thread lake~~ fix debezium reader thread lake Jul 10, 2022

ruanhang1993 mentioned this pull request Aug 1, 2022

[mysql] Release the debezium reader thread resources after reading finished #1192

Closed

ruanhang1993 reviewed Aug 1, 2022

View reviewed changes

Matrix42 force-pushed the #1350 branch from c222e55 to ea94851 Compare August 1, 2022 13:05

Matrix42 added 3 commits August 2, 2022 13:32

fix SnapshotSplitReader and BinlogSplitReader thread lake

b631183

fix checkstyle

1ef5799

fix review issue

c1e3c93

Matrix42 force-pushed the #1350 branch from c94f509 to c1e3c93 Compare August 2, 2022 05:32

Matrix42 requested a review from ruanhang1993 August 2, 2022 06:38

leonardBang reviewed Aug 2, 2022

View reviewed changes

fix review issue

ca86b68

leonardBang reviewed Aug 2, 2022

View reviewed changes

Matrix42 added 2 commits August 2, 2022 16:00

fix review issue

580bbca

fix review issue

087e619

leonardBang approved these changes Aug 2, 2022

View reviewed changes

leonardBang merged commit c96926b into apache:master Aug 2, 2022

ruanhang1993 mentioned this pull request Oct 17, 2022

[cdc-base] Release the debezium reader thread resources after reading finished #1619

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix debezium reader thread lake #1358

fix debezium reader thread lake #1358

Matrix42 commented Jul 10, 2022 •

edited

ruanhang1993 left a comment

ruanhang1993 Aug 1, 2022

ruanhang1993 Aug 1, 2022

ruanhang1993 Aug 1, 2022

Matrix42 commented Aug 1, 2022

Matrix42 commented Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

leonardBang Aug 2, 2022

Matrix42 Aug 2, 2022

leonardBang left a comment

Matrix42 commented Aug 2, 2022

	private static final long EXECUTOR_AWAIT_TERMINATION_TIMEOUT_SECONDS = 5;
	private static final long READER_CLOSE_TIMEOUT = 30L;

	"Failed to close the binlog split reader in {} ms.",
	"Failed to close the binlog split reader in {} seconds.",

	"Failed to close the snapshot split reader in {} ms.",
	"Failed to close the snapshot split reader in {} seconds.",

		assertNull(snapshotSplitReader.getExecutorService());
		assertTrue(snapshotSplitReader.getExecutorService().isTerminated());

fix debezium reader thread lake #1358

fix debezium reader thread lake #1358

Conversation

Matrix42 commented Jul 10, 2022 • edited

ruanhang1993 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matrix42 commented Aug 1, 2022

Matrix42 commented Aug 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leonardBang left a comment

Choose a reason for hiding this comment

Matrix42 commented Aug 2, 2022

Matrix42 commented Jul 10, 2022 •

edited