DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore #2543

ismailsimsek · 2021-07-16T21:03:52Z

https://issues.redhat.com/browse/DBZ-3621

jpechane · 2021-07-27T07:04:43Z

@ismailsimsek Thank you for the PR. Wouldn't it make more sense to add the stores into debezium-core module? In that case they will be usabe not only by server but by Debezium Engine too and it is also useful for it.

gunnarmorling

Some quick first comments. Thx!

debezium-server/debezium-server-core/pom.xml

...erver/debezium-server-core/src/main/java/io/debezium/server/storage/JdbcDatabaseHistory.java

...er/debezium-server-core/src/main/java/io/debezium/server/storage/JdbcOffsetBackingStore.java

ismailsimsek

Thank you @jpechane @gunnarmorling for the reviews, its ready for another review

debezium-core/pom.xml

ismailsimsek · 2021-08-09T08:55:45Z

debezium-core/src/main/java/io/debezium/storage/JdbcOffsetBackingStore.java

+        try {
+            log.debug("Saving data to state table...");
+            conn.setAutoCommit(false);
+            try (PreparedStatement sqlDelete = conn.prepareStatement(OFFSET_STORAGE_TABLE_DELETE);) {


looking at FileOffsetBackingStore i believe save method doing override?

should we put some effort to support saving offsets by multiple, different debezium processes? (supporting multiple databases)

I'm not quite following; the offset store would only ever be invoked in the context of single connectors.

i mean if there is more than one replication running, (2 jvm independent processes replicating mysql.database_1 and oracle.database_2) could we use same jdbc table for both replications as a FileOffsetBackingStore? would that be good feature to add?

one option could be reading table name from application.properties so user could provide different table name per replication

ismailsimsek · 2021-08-09T09:24:59Z

debezium-core/src/test/java/io/debezium/relational/history/JdbcDatabaseHistoryTest.java

+        assertThat(tables.size() == 1);
+        assertThat(tables.forTable(tableId).equals(table));
+        history.record(source, position2, historyRecord2.databaseName(), historyRecord2.schemaName(), historyRecord2.ddl(), tableChanges2);
+        history.record(source, position2, historyRecord2.databaseName(), historyRecord2.schemaName(), historyRecord2.ddl(), tableChanges2);


not sure why recording it once not returning table2? had to call it twice to make test pass

debezium-core/src/main/java/io/debezium/relational/history/JdbcDatabaseHistory.java

gunnarmorling

@ismailsimsek, a few more comments, we're getting there. Could you also look into adding a documentation page, probably something under https://debezium.io/documentation/reference/1.7/.

@jpechane, if you have specific preferences on how this should be represented in user docs, please speak up :)

Thanks a lot, @ismailsimsek!

debezium-core/pom.xml

debezium-core/src/main/java/io/debezium/relational/history/JdbcDatabaseHistory.java

debezium-core/src/main/java/io/debezium/storage/JdbcOffsetBackingStore.java

gunnarmorling · 2021-08-17T11:06:47Z

debezium-core/src/main/java/io/debezium/storage/JdbcOffsetBackingStore.java

+        try {
+            log.debug("Saving data to state table...");
+            conn.setAutoCommit(false);
+            try (PreparedStatement sqlDelete = conn.prepareStatement(OFFSET_STORAGE_TABLE_DELETE);) {


I'm not quite following; the offset store would only ever be invoked in the context of single connectors.

jpechane · 2021-08-17T11:22:29Z

@gunnarmorling IMHO This needs either a separate page or a section in Debezium Engine. The page/section should describe all implementations that are now on the plate - file-based, Kafka and JDBC.
If this is a separate page then there should be links from Debezium Engine and Debezium Server pages to it.

gunnarmorling

@ismailsimsek, took another look, please see my comments inline. I've also pushed a commit with some stylistic changes I did along the way.

debezium-core/src/main/java/io/debezium/relational/history/JdbcDatabaseHistory.java

debezium-embedded/src/main/java/io/debezium/embedded/storage/JdbcOffsetBackingStore.java

debezium-core/src/test/java/io/debezium/relational/history/JdbcDatabaseHistoryTest.java

wingerx · 2021-11-23T08:19:29Z

Hey, have any plans to merge this pr?

tomsej · 2021-12-16T20:07:54Z

Hi, @gunnarmorling I am also curious about the state of this PR. This feature would really help us. Thanks a lot!!

gunnarmorling · 2021-12-16T22:27:37Z

Thanks for the push, @tomsej and @wingerx. We can certainly try to revive this for 1.9. It has been a while since I last looked into this PR, so I'll need to refresh my memory first. There were some problems with unwanted dependencies established by this, not sure whether they have been sorted out by now, or not.

jpechane · 2022-07-12T15:35:06Z

@ismailsimsek @tomsej @wingerx Hi, I'd like to get this one for 2.0. I've merged the PR that provides modularized offset and history storages. Would any of you be willing to retrofit to it? Thanks

…ations DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore implementations

jpechane · 2023-01-10T07:56:54Z

@ismailsimsek Please check #4005 to see the plugins needed in the pom file to get the assembly built. Try it locally and then please fine tune deps to make sure that Debezium modules and deps not needed by the storage module are not packed in.

jpechane · 2023-01-12T08:11:57Z

@ismailsimsek The S3 PR has been merged. Do you think you'd be able to find few spare cycles so we can get this one out too?

subkanthi · 2023-01-13T00:00:07Z

The S3 PR has been merged. Do you think you'd be able to find few spare cycles so we can get this one out too?
@jpechane , happy to help with this PR in addressing the comments.

subkanthi · 2023-01-13T14:51:15Z

The S3 PR has been merged. Do you think you'd be able to find few spare cycles so we can get this one out too?
@jpechane , happy to help with this PR in addressing the comments.

@ismailsimsek , any ideas on how I can take over this work, only way I know is to create another PR from my fork. Guessing I wont have permission to commit to this branch. Thanks

subkanthi · 2023-01-13T15:03:24Z

The S3 PR has been merged. Do you think you'd be able to find few spare cycles so we can get this one out too?
@jpechane , happy to help with this PR in addressing the comments.

@ismailsimsek , any ideas on how I can take over this work, only way I know is to create another PR from my fork. Guessing I wont have permission to commit to this branch. Thanks

Thanks, got the invite to your repo.

github-actions · 2023-01-13T18:23:38Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2023-01-13T18:42:55Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

…figurable.

github-actions · 2023-01-16T16:59:14Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

…gurable.

…imsek/debezium into dbz_server_jdbc_storage

github-actions · 2023-01-16T17:02:23Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2023-01-16T21:17:33Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2023-01-18T19:28:56Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2023-01-19T15:28:13Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

…z_server_jdbc_storage

github-actions · 2023-01-19T22:01:32Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2023-01-19T22:01:40Z

Hi @ismailsimsek, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

jpechane · 2023-01-23T06:31:08Z

@ismailsimsek @subkanthi I cleaned the PR a bit and removed the merge commits, please let's continue at #4212

gunnarmorling requested changes Aug 3, 2021

View reviewed changes

ismailsimsek force-pushed the dbz_server_jdbc_storage branch 3 times, most recently from c9cf74e to b2521eb Compare August 8, 2021 07:39

ismailsimsek commented Aug 9, 2021

View reviewed changes

ismailsimsek marked this pull request as ready for review August 9, 2021 09:44

ismailsimsek changed the title ~~DBZ-3621 wip JdbcDatabaseHistory, JdbcOffsetBackingStore~~ DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore Aug 9, 2021

ismailsimsek requested a review from gunnarmorling August 10, 2021 19:45

ismailsimsek force-pushed the dbz_server_jdbc_storage branch from b2521eb to bfb9e24 Compare August 10, 2021 19:55

gunnarmorling reviewed Aug 17, 2021

View reviewed changes

gunnarmorling reviewed Aug 24, 2021

View reviewed changes

ismailsimsek force-pushed the dbz_server_jdbc_storage branch 3 times, most recently from 9428d21 to 01a6687 Compare August 25, 2021 16:27

ismailsimsek force-pushed the dbz_server_jdbc_storage branch 4 times, most recently from 9e450e7 to 9530c5a Compare September 18, 2021 08:32

ismailsimsek requested a review from gunnarmorling September 18, 2021 08:32

ismailsimsek force-pushed the dbz_server_jdbc_storage branch from 9530c5a to 922cdf5 Compare October 18, 2021 17:53

ismailsimsek force-pushed the dbz_server_jdbc_storage branch from 922cdf5 to 04683f8 Compare November 5, 2021 16:35

ismailsimsek force-pushed the dbz_server_jdbc_storage branch 3 times, most recently from 21f3a5c to eefb62a Compare July 17, 2022 09:16

DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore implement…

797d208

…ations DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore implementations

ismailsimsek force-pushed the dbz_server_jdbc_storage branch from 691d892 to 797d208 Compare January 9, 2023 13:47

subkanthi force-pushed the dbz_server_jdbc_storage branch from 797d208 to 0c5e4f0 Compare January 13, 2023 18:23

Merge branch 'main' into dbz_server_jdbc_storage_new

ba85aca

[DBZ-3621] Updated maven to include profiles, offset table is now con…

a371492

…figurable.

subkanthi added 2 commits January 16, 2023 12:01

DBZ-3621 Updated maven to include profiles, offset table is now confi…

3d02f18

…gurable.

DBZ-3621 Merge branch 'dbz_server_jdbc_storage' of github.com:ismails…

34711d0

…imsek/debezium into dbz_server_jdbc_storage

DBZ-3621 Fixed JdbcOffsetBackingStoreTest and fixed warnings

14081bf

DBZ-3621 Added integration test for JDBC offset storage.

944315a

Merge branch 'debezium:main' into dbz_server_jdbc_storage

1921e23

subkanthi added 3 commits January 19, 2023 17:01

DBZ-3621 Added integration test for JDBC offset storage.

e72a6b4

Merge remote-tracking branch 'origin/dbz_server_jdbc_storage' into db…

7980eb5

…z_server_jdbc_storage

DBZ-3621 Added README.md

0ac219f

jpechane closed this Jan 23, 2023

subkanthi mentioned this pull request Apr 7, 2023

DBZ-3621 Saving Debezium offsets in JDBC database #4438

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore #2543

DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore #2543

ismailsimsek commented Jul 16, 2021 •

edited by gunnarmorling

jpechane commented Jul 27, 2021

gunnarmorling left a comment

ismailsimsek left a comment

ismailsimsek Aug 9, 2021

gunnarmorling Aug 17, 2021

ismailsimsek Aug 18, 2021

ismailsimsek Aug 9, 2021

gunnarmorling left a comment

gunnarmorling Aug 17, 2021

jpechane commented Aug 17, 2021

gunnarmorling left a comment

wingerx commented Nov 23, 2021

tomsej commented Dec 16, 2021

gunnarmorling commented Dec 16, 2021

jpechane commented Jul 12, 2022

jpechane commented Jan 10, 2023

jpechane commented Jan 12, 2023

subkanthi commented Jan 13, 2023

subkanthi commented Jan 13, 2023

subkanthi commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

jpechane commented Jan 23, 2023

DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore #2543

DBZ-3621 adding JdbcDatabaseHistory, JdbcOffsetBackingStore #2543

Conversation

ismailsimsek commented Jul 16, 2021 • edited by gunnarmorling

jpechane commented Jul 27, 2021

gunnarmorling left a comment

Choose a reason for hiding this comment

ismailsimsek left a comment

Choose a reason for hiding this comment

ismailsimsek Aug 9, 2021

Choose a reason for hiding this comment

gunnarmorling Aug 17, 2021

Choose a reason for hiding this comment

ismailsimsek Aug 18, 2021

Choose a reason for hiding this comment

ismailsimsek Aug 9, 2021

Choose a reason for hiding this comment

gunnarmorling left a comment

Choose a reason for hiding this comment

gunnarmorling Aug 17, 2021

Choose a reason for hiding this comment

jpechane commented Aug 17, 2021

gunnarmorling left a comment

Choose a reason for hiding this comment

wingerx commented Nov 23, 2021

tomsej commented Dec 16, 2021

gunnarmorling commented Dec 16, 2021

jpechane commented Jul 12, 2022

jpechane commented Jan 10, 2023

jpechane commented Jan 12, 2023

subkanthi commented Jan 13, 2023

subkanthi commented Jan 13, 2023

subkanthi commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

jpechane commented Jan 23, 2023

ismailsimsek commented Jul 16, 2021 •

edited by gunnarmorling