Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL DATETIME values under year 100 are converted incorrectly to ClickHouse's DateTime64. #470

Closed
Selfeer opened this issue Feb 26, 2024 · 3 comments

Comments

@Selfeer
Copy link
Collaborator

Selfeer commented Feb 26, 2024

In MySQL it is possible to create tables with DATETIME values under year 1000. When creating a MySQL table with DATETIME column and inserting values with years1-99 these values are not converted to minimal DateTime64 value in ClickHouse.

For example.

In MySQL,

CREATE TABLE test (date DATETIME(5)) ENGINE = InnoDB;

INSERT INTO test VALUES ('0001-01-01 00:00:00.00000');
INSERT INTO test VALUES ('999-01-01 00:00:00.00000');
INSERT INTO test VALUES ('99-01-01 00:00:00.00000');

in ClickHouse we get,

SELECT * FROM test;

┌─my_row_id─┬──────────────────────date─┬────────────_version─┬─is_deleted─┐
│         12001-01-01 00:00:00.0000017621643536598303550 │
│         21900-01-01 00:00:00.0000017621644123800863560 │
│         31999-01-01 00:00:00.0000017621647982560543630 │
└───────────┴───────────────────────────┴─────────────────────┴────────────┘

If in MySQL we insert values above 100 the result is as expected,

INSERT INTO test VALUES ('100-01-01 00:00:00.00000');

In ClickHouse,

SELECT * FROM test WHERE my_row_id == 4;

┌─my_row_id─┬──────────────────────date─┬────────────_version─┬─is_deleted─┐
│        41900-01-01 00:00:00.0000017621681495049503700 │
└───────────┴───────────────────────────┴─────────────────────┴────────────┘

Expected result:

These values should be automatically converted to DateTime64 minimal value which is 1900-01-01 00:00:00

@subkanthi
Copy link
Collaborator

image
image

The value thats returned by debezium is 915148800000000 for 0099-01-01 00:00:00.000000 which is 1999-01-01T00:00:00Z

@subkanthi
Copy link
Collaborator

subkanthi commented Feb 26, 2024

https://stackoverflow.com/questions/73956216/datetime-from-mysql-in-kafka-is-converted-to-wrong-timestamp-for-dates-less-tha

[enable.time.adjuster](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-enable-time-adjuster)

true

Boolean value that indicates whether the connector converts a 2-digit year specification to 4 digits. Set to false when conversion is fully delegated to the database.

MySQL allows users to insert year values with either 2-digits or 4-digits. For 2-digit values, the value gets mapped to a year in the range 1970 - 2069. The default behavior is that the connector does the conversion.

@Selfeer
Copy link
Collaborator Author

Selfeer commented Feb 27, 2024

Using the Debezium parameter "enable.time.adjuster" : "false", seems to resolve the problem.

@Selfeer Selfeer closed this as completed Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants