fix(sink): switch remote sink chrono types to java.time.* #14413

xuefengze · 2024-01-08T08:27:03Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

resolve #13684 and fix #14198

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

cassandra-sink:
Before:
timestamp is allowed to sink into timestamp type in cassandra.

After:
timestamp cannot be sinked and has to be converted to timestamptz (via AT TIME ZONE) inside RisingWave before sinking out.

src/jni_core/src/lib.rs

xiangjinwu

Thanks for the fix!

Also spotted some other references to the legacy classes:

connector-node/risingwave-connector-service/src/main/java/com/risingwave/connector/JsonDeserializer.java
connector-node/risingwave-sink-mock-flink/risingwave-sink-mock-flink-runtime/src/main/java/com/risingwave/mock/flink/runtime/RowDataImpl.java
connector-node/risingwave-sink-deltalake/src/main/java/com/risingwave/connector/DeltaLakeSink.java
connector-node/risingwave-source-cdc/src/main/java/com/risingwave/connector/cdc/debezium/converters/DatetimeTypeConverter.java

After cleaning these up, is it possible to add a static analysis rule to disallow usage of them? We have this in rust:

risingwave/clippy.toml

Lines 1 to 2 in df1b9de

    
           disallowed-methods = [ 
        
               { path = "std::iter::Iterator::zip", reason = "Please use `zip_eq_fast` if it's available. Otherwise use `zip_eq_debug`" },

src/jni_core/src/lib.rs

xiangjinwu · 2024-01-10T07:35:39Z

...tor-node/risingwave-sink-cassandra/src/main/java/com/risingwave/connector/CassandraUtil.java

            case TIMESTAMP:
+                return ((LocalDateTime) value).toInstant(ZoneOffset.UTC);


https://cassandra.apache.org/doc/stable/cassandra/cql/types.html#timestamps

Similar to ClickHouse, there is no naive datetime type in Cassandra. And when interpreting one without timezone:

The time zone may be omitted if desired ('2011-02-03 04:05:00'), and if so, the date will be interpreted as being in the time zone under which the coordinating Cassandra node is configured. There are however difficulties inherent in relying on the time zone configuration being as expected, so it is recommended that the time zone always be specified for timestamps when feasible.

To be fully correct, we either need to use the Cassandra coordinator node time zone (rather than hardcoding to UTC here), or reject sinking and require user to AT TIME ZONE 'xxx' to get a timestamptz before sink.

...risingwave-connector-test/src/test/java/com/risingwave/connector/sink/jdbc/JDBCSinkTest.java

StrikeW

Would you mind adding e2e test to cover the case of #14198 in e2e_test/sink/remote/jdbc.load.slt?

...risingwave-connector-test/src/test/java/com/risingwave/connector/sink/jdbc/JDBCSinkTest.java

xuefengze · 2024-01-11T06:46:02Z

mysql-sink: When insert 2038-01-19 03:14:07.499999Z to mysql timestamp type, will get error:

message Data truncation: Incorrect datetime value: '2038-01-19 11:14:07.499999' (my local timezone +8)

It appears that mysql jdbc will convert it to the default timezone? Maybe we can cast offsetdatetime to localdatetime before write to solve this issue?
cc @xiangjinwu

https://github.com/mysql/mysql-connector-j/blob/release/8.x/src/main/protocol-impl/java/com/mysql/cj/protocol/a/OffsetDateTimeValueEncoder.java#L72-L83

xiangjinwu · 2024-01-11T07:24:21Z

my local timezone +8

It is better to set the session/connection timezone to UTC when interacting with MySQL (and other systems as well). Local times are problematic and shall be avoided whenever possible.

https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-known-issues-limitations.html

When Connector/J retrieves timestamps for a daylight saving time (DST) switch day using the getTimeStamp() method on the result set, some of the returned values might be wrong. In order to avoid such errors, we recommend setting a connection time zone that uses a monotonic clock by, for example, setting connectionTimeZone=UTC, and configuring other date-time connection properties according to your needs; see Section 6.6, “Handling of Date-Time Values” for details.

wenym1

Rest LGTM! Thanks for the PR

wenym1 · 2024-01-12T05:11:53Z

e2e_test/sink/remote/jdbc.check.pg.slt

@@ -3,17 +3,17 @@
 query I
 select * from t_remote_0 order by id;
 ----
-1 Alex 28208 281620391 4986480304337356659 28162.0391 2.03 28162.0391 2023-03-20 10:18:30


Any reason to change from a normal datetime to a strange one?

Would you mind adding e2e test to cover the case of #14198 in e2e_test/sink/remote/jdbc.load.slt?

cover the case of #14198.

xiangjinwu · 2024-01-12T03:52:33Z

...de/risingwave-connector-service/src/main/java/com/risingwave/connector/JsonDeserializer.java

+    private static LocalDate castDate(Object value) {
        try {
-            Long days = castLong(value) - 1;
-            return Date.valueOf(LocalDate.of(1, 1, 1).plusDays(days));
+            Long days = castLong(value);
+            return LocalDate.ofEpochDay(days);


old: 1 -> 0001-01-01

new: 1 -> 1970-01-02

Look like related to #13486. Was the fix incomplete until this change also merged?

Was the fix incomplete until this change also merged?

No.
It seems that there are no remote sinks using JsonDeserializer currently? So I change the value to the number of days since epoch.

...de/risingwave-connector-service/src/main/java/com/risingwave/connector/JsonDeserializer.java

xiangjinwu · 2024-01-12T05:15:53Z

...tor-node/risingwave-sink-deltalake/src/main/java/com/risingwave/connector/DeltaLakeSink.java

-                            values = ((Timestamp) row.get(i)).getTime();
-                        } else if (row.get(i) instanceof java.sql.Date) {
+                        if (row.get(i) instanceof LocalDateTime) {
                            values =
-                                    ChronoUnit.DAYS.between(
-                                            LocalDate.ofEpochDay(0),
-                                            ((java.sql.Date) row.get(i)).toLocalDate());
+                                    ((LocalDateTime) row.get(i))
+                                            .toInstant(ZoneOffset.UTC)
+                                            .toEpochMilli();


cc @xxhZs regarding #13420 #13677 I am not sure whether we need to test for all 4 LocalDate/LocalTime/LocalDateTime/OffsetDateTime out of Object ...

https://github.com/delta-io/delta/blob/master/PROTOCOL.md#timestamp-without-timezone-timestampntz
Also note that:

RIsingWave timestamp is DeltaLake TimestampNtz (isAdjustedToUTC=false)

RisingWave timestamptz is DeltaLake Timestamp (isAdjustedToUTC=true)

Co-authored-by: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>

xiangjinwu · 2024-01-12T06:27:14Z

After cleaning these up, is it possible to add a static analysis rule to disallow usage of them?

I figured this out: (Still possible to use Date/Timestamp from java.sql.* until we stop suppressing AvoidStarImport.)

diff --git a/java/tools/maven/checkstyle.xml b/java/tools/maven/checkstyle.xml
index 4bd0d510e0..f681633272 100644
--- a/java/tools/maven/checkstyle.xml
+++ b/java/tools/maven/checkstyle.xml
@@ -191,10 +191,24 @@ This file is based on the checkstyle file of Apache Beam.
             <property name="illegalPkgs" value="io.netty"/>
             <message key="import.illegal" value="{0}; Use flink-shaded-netty instead."/>
         </module>
+        <!--
         <module name="IllegalImport">
             <property name="illegalPkgs" value="com.google.common"/>
             <message key="import.illegal" value="{0}; Use flink-shaded-guava instead."/>
         </module>
+        -->
+        <module name="IllegalImport">
+            <property name="illegalClasses" value="java.util.Date, java.sql.Date, java.sql.Timestamp, java.sql.Time"/>
+            <message key="import.illegal" value="{0}; Use java.time.* instead."/>
+        </module>
+        <module name="IllegalImport">
+            <property name="illegalClasses" value="java.text.DateFormat, java.text.SimpleDateFormat"/>
+            <message key="import.illegal" value="{0}; Use java.time.* instead."/>
+        </module>
+        <module name="IllegalImport">
+            <property name="illegalClasses" value="java.util.Calendar, java.util.GregorianCalendar"/>
+            <message key="import.illegal" value="{0}; Use java.time.* instead."/>
+        </module>
 
         <module name="RedundantModifier">
             <!-- Checks for redundant modifiers on various symbol definitions.
diff --git a/java/tools/maven/suppressions.xml b/java/tools/maven/suppressions.xml
index 5e19f59eae..11b4774359 100644
--- a/java/tools/maven/suppressions.xml
+++ b/java/tools/maven/suppressions.xml
@@ -30,5 +30,4 @@ under the License.
 
     <suppress files=".*" checks="javadoc"/>
     <suppress files=".*" checks="AvoidStarImport"/>
-    <suppress files=".*" checks="IllegalImport"/>
 </suppressions>
\ No newline at end of file

Co-authored-by: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>

xuefengze added 4 commits January 8, 2024 16:06

fix(sink): switch remote sink chrono types to java.time.*

7c6af7f

fix test

320bd8c

fix

2d60b8e

fix

0714a3b

xuefengze marked this pull request as ready for review January 9, 2024 02:25

xuefengze requested review from wenym1 and xiangjinwu January 9, 2024 02:25

wenym1 requested review from StrikeW and xxhZs January 9, 2024 08:40

wenym1 reviewed Jan 9, 2024

View reviewed changes

src/jni_core/src/lib.rs Outdated Show resolved Hide resolved

xiangjinwu approved these changes Jan 10, 2024

View reviewed changes

StrikeW reviewed Jan 10, 2024

View reviewed changes

...risingwave-connector-test/src/test/java/com/risingwave/connector/sink/jdbc/JDBCSinkTest.java Outdated Show resolved Hide resolved

...risingwave-connector-test/src/test/java/com/risingwave/connector/sink/jdbc/JDBCSinkTest.java Outdated Show resolved Hide resolved

wenym1 added ci/run-java-binding-tests ci/run-java-connector-node-integration-tests ci/run-connector-node-integration-tests ci/run-e2e-cassandra-sink-tests labels Jan 10, 2024

xuefengze added 4 commits January 11, 2024 16:34

update

560ac29

fix

3a6ba94

fix

d057497

fix

46bd607

wenym1 approved these changes Jan 12, 2024

View reviewed changes

xiangjinwu reviewed Jan 12, 2024

View reviewed changes

fix typo

cdb0c91

Co-authored-by: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>

update java checkstyle

90db6af

xuefengze added breaking-change type/fix Bug fix labels Jan 12, 2024

xiangjinwu approved these changes Jan 12, 2024

View reviewed changes

xuefengze added this pull request to the merge queue Jan 15, 2024

Merged via the queue into risingwavelabs:main with commit f6892e9 Jan 15, 2024
37 of 38 checks passed

xuefengze deleted the replace_java_time branch January 15, 2024 06:48

Little-Wallace pushed a commit that referenced this pull request Jan 20, 2024

fix(sink): switch remote sink chrono types to java.time.* (#14413)

09c5729

Co-authored-by: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>

This was referenced Jan 22, 2024

test(connector): add cassandra/scyllabd-sink compatible test #14709

Merged

update cassandra-sink data type mapping risingwavelabs/risingwave-docs#1798

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(sink): switch remote sink chrono types to java.time.* #14413

fix(sink): switch remote sink chrono types to java.time.* #14413

xuefengze commented Jan 8, 2024 •

edited by lmatz

Loading

xiangjinwu left a comment

xiangjinwu Jan 10, 2024 •

edited

Loading

StrikeW left a comment

xuefengze commented Jan 11, 2024 •

edited

Loading

xiangjinwu commented Jan 11, 2024 •

edited

Loading

wenym1 left a comment

wenym1 Jan 12, 2024

xuefengze Jan 12, 2024

xiangjinwu Jan 12, 2024

xuefengze Jan 12, 2024 •

edited

Loading

xiangjinwu Jan 12, 2024

xuefengze Jan 12, 2024

xiangjinwu commented Jan 12, 2024 •

edited

Loading

	disallowed-methods = [
	{ path = "std::iter::Iterator::zip", reason = "Please use `zip_eq_fast` if it's available. Otherwise use `zip_eq_debug`" },

		case TIMESTAMP:
		return ((LocalDateTime) value).toInstant(ZoneOffset.UTC);

fix(sink): switch remote sink chrono types to java.time.* #14413

fix(sink): switch remote sink chrono types to java.time.* #14413

Conversation

xuefengze commented Jan 8, 2024 • edited by lmatz Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

xiangjinwu left a comment

Choose a reason for hiding this comment

xiangjinwu Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

StrikeW left a comment

Choose a reason for hiding this comment

xuefengze commented Jan 11, 2024 • edited Loading

xiangjinwu commented Jan 11, 2024 • edited Loading

wenym1 left a comment

Choose a reason for hiding this comment

wenym1 Jan 12, 2024

Choose a reason for hiding this comment

xuefengze Jan 12, 2024

Choose a reason for hiding this comment

xiangjinwu Jan 12, 2024

Choose a reason for hiding this comment

xuefengze Jan 12, 2024 • edited Loading

Choose a reason for hiding this comment

xiangjinwu Jan 12, 2024

Choose a reason for hiding this comment

xuefengze Jan 12, 2024

Choose a reason for hiding this comment

xiangjinwu commented Jan 12, 2024 • edited Loading

xuefengze commented Jan 8, 2024 •

edited by lmatz

Loading

xiangjinwu Jan 10, 2024 •

edited

Loading

xuefengze commented Jan 11, 2024 •

edited

Loading

xiangjinwu commented Jan 11, 2024 •

edited

Loading

xuefengze Jan 12, 2024 •

edited

Loading

xiangjinwu commented Jan 12, 2024 •

edited

Loading