Improve UX of the CDC API #21536

TomaszGaweda · 2022-06-03T19:19:52Z

Various UX Changes for Debezium integration:

ChangeRecord fields are no longer "lazy" - values were there, just embedded in the json file. Now it's extracted directly.
old value and new value of change record are available. Previously user had only generic value, which was only new value for insert, sync and update and old value for delete. It was impossible to check e.g. which field had changed.
connector method to make code less stringly type - pass directly the connector class instance.

Breaking changes (list specific methods/types/messages):

API - type of ParsingException - now it's RuntimeException. Changed also ChangeRecord, so that it doesn't declare to throw this exception when not needed anymore.
serialized form - serialized form of ChangeRecord

Checklist:

Labels (Team:, Type:, Source:, Module:) and Milestone set
Label Add to Release Notes or Not Release Notes content set
Request reviewers if possible
Send backports/forwardports if fix needs to be applied to past/future releases
New public APIs have @Nonnull/@Nullable annotations
New public APIs have @since tags in Javadoc

hz-devops-test · 2022-06-06T18:38:40Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 3 errors reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:46:46: Using the '.*' form of import should be avoided - com.hazelcast.jet.cdc.Operation.*. [AvoidStarImport]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:283:45: '(' is followed by whitespace. [ParenPad]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:338: Line is longer than 123 characters (found 144). [LineLength]
--------------------------

hz-devops-test · 2022-06-06T18:41:07Z

The job Hazelcast-pr-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 3 errors reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:46:46: Using the '.*' form of import should be avoided - com.hazelcast.jet.cdc.Operation.*. [AvoidStarImport]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:283:45: '(' is followed by whitespace. [ParenPad]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:338: Line is longer than 123 characters (found 144). [LineLength]
--------------------------

hz-devops-test · 2022-06-06T19:05:01Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   CdcSinksTest.deleteFromLocalMap_ViaValueProjection:168->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.deleteFromRemoteMap:157->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.deleteFromRemoteMap_ViaValueProjection:200->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.deleteWithoutInsertNorUpdate:270->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.deleteWithoutInsertNorUpdate:270->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.insertIntoRemoteMap:103->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.reordering:226->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.reordering:226->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[INFO] 
[ERROR]   CdcSinksTest.reordering_differentIds:259->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[INFO] --- maven-failsafe-plugin:2.22.2:integration-test (default) @ hazelcast-jet-protobuf ---
[ERROR]   CdcSinksTest.reordering_differentIds:259->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.reordering_insertDelete:248->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.reordering_insertDelete:248->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.reordering_syncUpdate:237->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.sourceSwitch:286->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.updateLocalMap:114->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.updateRemoteMap:130->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[ERROR]   CdcSinksTest.updateRemoteMap:130->PipelineTestSupport.execute:82->PipelineTestSupport.execute:86->TestInClusterSupport.execute:122 ? Completion
[INFO] 
[ERROR] Tests run: 28, Failures: 0, Errors: 17, Skipped: 0
[INFO] 
[ERROR] There are test failures.

hz-devops-test · 2022-06-07T15:08:45Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   MySqlCdcIntegrationTest.customers:91->HazelcastTestSupport.assertEqualsEventually:1005->HazelcastTestSupport.assertTrueEventually:1328->HazelcastTestSupport.assertTrueEventually:1226->HazelcastTestSupport.lambda$assertEqualsEventually$8:1005 expected:<4> but was:<0>
[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.

hz-devops-test · 2022-06-07T15:24:07Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordCdcSourceP.java:31:15: Unused import - com.hazelcast.jet.impl.util.ExceptionUtil.isRestartableException. [UnusedImports]
--------------------------

…ere copied to json by transform, now we just read them directly

hz-devops-test · 2022-06-07T15:34:55Z

The job Hazelcast-pr-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordCdcSourceP.java:31:15: Unused import - com.hazelcast.jet.impl.util.ExceptionUtil.isRestartableException. [UnusedImports]
--------------------------

hz-devops-test · 2022-06-07T19:13:41Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-postgres: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/extensions/cdc-postgres/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder/extensions/cdc-postgres/src/main/java/com/hazelcast/jet/cdc/postgres/PostgresCdcSources.java:498:5: Missing a Javadoc comment. [MissingJavadocType]
--------------------------

hz-devops-test · 2022-06-07T19:16:24Z

The job Hazelcast-pr-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-postgres: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-postgres/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-postgres/src/main/java/com/hazelcast/jet/cdc/postgres/PostgresCdcSources.java:498:5: Missing a Javadoc comment. [MissingJavadocType]
--------------------------

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java

extensions/cdc-postgres/src/main/java/com/hazelcast/jet/cdc/postgres/PostgresCdcSources.java

extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/DebeziumCdcIntegrationTest.java

frant-hartm · 2022-06-09T08:08:15Z

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/RecordPartImpl.java

 import java.util.Map;
 import java.util.Objects;

-class RecordPartImpl implements RecordPart {
+class RecordPartImpl implements RecordPart, Serializable {
+    private static final long serialVersionUID = 1L;


Why does it need to be Serializable?

No longer needed, connected to CdcSinksTest - now it's done on suppliers there and Serializable is not necessary

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/PropertyRules.java

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java

frant-hartm · 2022-06-09T08:17:29Z

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java

+        try {
+            switch (operation()) {
+                case SYNC:
+                case INSERT:
+                case UPDATE: return value();
+                default: return null;
+            }
+        } catch (ParsingException e) {
+            throw rethrow(e);
+        }
+    }


Why do you have the default implementation? There is only 1 implementation of the interface, and it overrides this method.

In case if user implemented this own ChangeRecord. He shouldn't really, but it wasn't constrained in any way. If we don't consider it a public api, then I'll remove it.

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordImpl.java

frant-hartm · 2022-06-09T08:22:14Z

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordCdcSourceP.java

+        String oldValueJson = Values.convertToString(valueSchema.field("before").schema(), value.get("before"));
+        String newValueJson = Values.convertToString(valueSchema.field("after").schema(), value.get("after"));


We should try to reduce the amount of garbage created - not all will use both before & after, not all fields will be read,
I wonder if this can be improved, e.g. by converting lazily, or not converting at all if not needed.

I may save the json value and convert lazily using some generic JSON mapper (jackson/gson). IIRC schema is not even requried, so this should not be a problem

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordImpl.java

Not using double-checked locking for simplicity, such values should be retrieved from one thread only

hz-devops-test · 2022-06-09T15:01:15Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 3 errors reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_3/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_3/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java:24:15: Unused import - com.hazelcast.jet.impl.util.ExceptionUtil.rethrow. [UnusedImports]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_3/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordCdcSourceP.java:72: Line is longer than 123 characters (found 139). [LineLength]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_3/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordCdcSourceP.java:73: Line is longer than 123 characters (found 136). [LineLength]
--------------------------

hz-devops-test · 2022-06-09T15:32:33Z

The job Hazelcast-pr-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project hazelcast-jet-cdc-debezium: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There are 14 errors reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordImpl.java:26:8: Unused import - java.io.Serializable. [UnusedImports]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/RecordPartImpl.java:25:8: Unused import - java.io.Serializable. [UnusedImports]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:215:19: '->' is not followed by whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:215:19: '->' is not preceded with whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:217:19: '->' is not followed by whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:217:19: '->' is not preceded with whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:219:19: '->' is not followed by whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:219:19: '->' is not preceded with whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:221:19: '->' is not followed by whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:221:19: '->' is not preceded with whitespace. [WhitespaceAround]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/CdcSinksTest.java:344: Line is longer than 123 characters (found 124). [LineLength]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/impl/PropertyRulesTest.java:1: File does not end with a newline. [NewlineAtEndOfFile]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/impl/PropertyRulesTest.java:1: Line does not match expected header line of '/*'. [Header]
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/impl/PropertyRulesTest.java:9:31: Using the '.*' form of import should be avoided - org.junit.Assert.*. [AvoidStarImport]
--------------------------

Co-authored-by: František Hartman <frant.hartm@gmail.com>

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java

ldziedziul · 2022-06-14T09:29:50Z

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ParsingException.java

@@ -24,7 +24,7 @@
 * @since Jet 4.2
 */
 @EvolvingApi
-public class ParsingException extends Exception {
+public class ParsingException extends RuntimeException {


Why this change?

Most of the operations now are not lazy-parsing the json, so no exception is really expected. However, if user already used methods like .database(), he may wrap the invocation in the try-catch block. Therefore, I cannot completely remove "throws ParsingException" from those methods to not break source compatibility, yet new users should not be required to deal with exception that should never occurs. One workaround is to change ParsingException to RuntimeException - user by default won't deal with it, but old code will still work

I wonder if we should mark this class as deprecated then?

It is still used in lazy-parsing in RecordPartImpl, although I'm not against changing this to other exception type.

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/impl/ChangeRecordImpl.java

ldziedziul · 2022-06-14T10:02:26Z

extensions/cdc-debezium/src/test/java/com/hazelcast/jet/cdc/impl/PropertyRulesTest.java

+import static org.junit.Assert.fail;
+
+@Category(QuickTest.class)
+public class PropertyRulesTest {


ldziedziul · 2022-06-14T10:08:08Z

extensions/cdc-postgres/src/main/java/com/hazelcast/jet/cdc/postgres/PostgresCdcSources.java

+        @Nonnull
+        public Builder setSnapshotMode(@Nonnull PostgresSnapshotMode snapshotMode) {
+            PostgresConnectorConfig.SnapshotMode debeziumMode;
+            switch (snapshotMode) {


I would prefer to stick with the IntelliJ default formatting to avoid whitespace changes when others will work on this piece of the code

Good point, done. I was too much inspired by new switch expressions ;P

…um-ux-changes

hz-devops-test · 2022-06-14T11:47:35Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ClientMessageProtectionTest.testExceededMessageSize:154->writeClientMessage:245 ? Socket
[INFO] 
[ERROR] Tests run: 4351, Failures: 0, Errors: 1, Skipped: 8
[INFO] 
[ERROR] There are test failures.

hz-devops-test · 2022-06-23T14:05:06Z

The job Hazelcast-pr-EE-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal on project hazelcast-sql: Could not resolve dependencies for project com.hazelcast:hazelcast-sql:jar:5.2-SNAPSHOT: Could not transfer artifact com.hazelcast.jsurfer:jsurfer-jackson-jr:jar:0.11-20220329.074450-11 from/to snapshot-repository (https://oss.sonatype.org/content/repositories/snapshots): Transfer failed for https://oss.sonatype.org/content/repositories/snapshots/com/hazelcast/jsurfer/jsurfer-jackson-jr/0.11-SNAPSHOT/jsurfer-jackson-jr-0.11-20220329.074450-11.jar 502 Bad Gateway -> [Help 1]
--------------------------

frant-hartm

LGTM, one minor thing to consider.

frant-hartm · 2022-06-14T06:34:30Z

extensions/cdc-debezium/src/main/java/com/hazelcast/jet/cdc/ChangeRecord.java

-     *
-     * @throws ParsingException if the timestamp field isn't present or
-     *                          is unparseable


The change record is marked with @EvolvingApi so we can remove the throws ParsingException if it's not the case anymore.

…onger true

hz-devops-test · 2022-06-23T14:44:05Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   Hz3SinksTest>BaseHz3Test.setUp:107 ? Guardian javax.net.ssl.SSLException: java...
[INFO] 
[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[ERROR] There are test failures.

TomaszGaweda added 2 commits June 3, 2022 21:15

New methods with less stringly-typed argument

9c038df

Ability to get old and new value in ChangeRecord

13c3ae8

TomaszGaweda added 3 commits June 7, 2022 17:33

No more pseudo-lazy values in the ChangeRecordImpl, that previously w…

30ee22c

…ere copied to json by transform, now we just read them directly

Get rid of old constructor

46e804a

Test simplification, cleanup

40c56bc

TomaszGaweda force-pushed the debezium-ux-changes branch from 7bdc5e2 to 40c56bc Compare June 7, 2022 15:33

TomaszGaweda self-assigned this Jun 7, 2022

TomaszGaweda added Type: Enhancement Source: Internal PR or issue was opened by an employee Add to Release Notes Module: Jet Issues/PRs for Jet Team: Integration labels Jun 7, 2022

TomaszGaweda marked this pull request as ready for review June 7, 2022 16:52

TomaszGaweda requested a review from frant-hartm June 7, 2022 16:52

First-class support for snapshotters

ca92caa

Checkstyle

310ed1b

frant-hartm suggested changes Jun 9, 2022

View reviewed changes

Make JSON parsing lazy

267fa1c

Not using double-checked locking for simplicity, such values should be retrieved from one thread only

TomaszGaweda added 3 commits June 9, 2022 17:07

ChangeRecord and RecordPartImpl not serializable, checkstyle

23e1d85

Test for property rules, fixed typo

f828229

Avoid API leak

4848a2e

TomaszGaweda and others added 3 commits June 9, 2022 19:18

Checkstyle

94f559c

Merge branch 'master' into debezium-ux-changes

c6a5248

CR suggestion

9b12103

Co-authored-by: František Hartman <frant.hartm@gmail.com>

ldziedziul reviewed Jun 14, 2022

View reviewed changes

TomaszGaweda added 2 commits June 14, 2022 12:58

Address CR comments

6873e0b

Merge remote-tracking branch 'origin/debezium-ux-changes' into debezi…

534d32d

…um-ux-changes

hazelcast deleted a comment from hz-devops-test Jun 15, 2022

TomaszGaweda requested review from frant-hartm and ldziedziul June 20, 2022 10:48

TomaszGaweda mentioned this pull request Jun 20, 2022

CDC real time analytics demo hazelcast/hazelcast-code-samples#540

Merged

ldziedziul approved these changes Jun 20, 2022

View reviewed changes

Missing API leakage fixed

6c442d9

frant-hartm approved these changes Jun 23, 2022

View reviewed changes

Addressed CR comment: get rid of "throws ParsingException" where no l…

c240949

…onger true

TomaszGaweda changed the title ~~Debezium UX Changes~~ CDC API Enhancements: expose new and old value of record, more strongly-typed API, less exceptions Jun 23, 2022

TomaszGaweda merged commit 8763a4a into hazelcast:master Jun 23, 2022

TomaszGaweda changed the title ~~CDC API Enhancements: expose new and old value of record, more strongly-typed API, less exceptions~~ Improve UX of the CDC API Jun 23, 2022

AyberkSorgun added this to the 5.2 milestone Jun 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve UX of the CDC API #21536

Improve UX of the CDC API #21536

TomaszGaweda commented Jun 3, 2022 •

edited

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

frant-hartm Jun 9, 2022

TomaszGaweda Jun 9, 2022

frant-hartm Jun 9, 2022

TomaszGaweda Jun 9, 2022

frant-hartm Jun 9, 2022

TomaszGaweda Jun 9, 2022

hz-devops-test commented Jun 9, 2022

hz-devops-test commented Jun 9, 2022

ldziedziul Jun 14, 2022

TomaszGaweda Jun 14, 2022

ldziedziul Jun 14, 2022

TomaszGaweda Jun 20, 2022

ldziedziul Jun 14, 2022

ldziedziul Jun 14, 2022

TomaszGaweda Jun 14, 2022

hz-devops-test commented Jun 14, 2022

hz-devops-test commented Jun 23, 2022

frant-hartm left a comment

frant-hartm Jun 14, 2022

TomaszGaweda Jun 23, 2022

hz-devops-test commented Jun 23, 2022

		String oldValueJson = Values.convertToString(valueSchema.field("before").schema(), value.get("before"));
		String newValueJson = Values.convertToString(valueSchema.field("after").schema(), value.get("after"));

Improve UX of the CDC API #21536

Improve UX of the CDC API #21536

Conversation

TomaszGaweda commented Jun 3, 2022 • edited

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 6, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

hz-devops-test commented Jun 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hz-devops-test commented Jun 9, 2022

hz-devops-test commented Jun 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hz-devops-test commented Jun 14, 2022

hz-devops-test commented Jun 23, 2022

frant-hartm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hz-devops-test commented Jun 23, 2022

TomaszGaweda commented Jun 3, 2022 •

edited