Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sink): fix starrocks doris and clickhouse decimal #15664

Merged
merged 6 commits into from
Apr 11, 2024

Conversation

xxhZs
Copy link
Contributor

@xxhZs xxhZs commented Mar 13, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Previously we would report an error when insert ck doris starrocks nan inf -inf, now we have made the change
in clickhouse ,if clickhouse is nullable, will insert null. If not, we will insert 0.
in doris/storrocks, if decimal is out of bounds or inf -inf nan, we insert null.

Due to the starrocks update, we removed some of the calibration logic for decimal

The reason for the change is that

  1. previously we needed precision to make precision corrections, e.g. if we had a 1.123456 and starrocks had a precision of 3, we would need to convert it to 1.123 first, but recent versions of starrocks have been able to convert it automatically, so it doesn't require us to do this (doris is still required)

  2. Regarding the maximum number of bits, as long as the maximum number of bits is exceeded, there is no way for us to insert into the downstream. Previously, we would judge the number of bits because we wanted to report the error ahead of time, but then we found out that there is no point in doing so, so we remove this logic

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Previously we would report an error when insert ck doris starrocks nan inf -inf, now we have made the change
in clickhouse ,if clickhouse is nullable, will insert null. If not, we will insert 0.

@github-actions github-actions bot added the type/fix Bug fix label Mar 13, 2024
@xxhZs xxhZs requested review from hzxa21, lmatz and wenym1 March 13, 2024 10:03
@xxhZs xxhZs added the user-facing-changes Contains changes that are visible to users label Mar 13, 2024
@wenym1
Copy link
Contributor

wenym1 commented Mar 13, 2024

in clickhouse ,if clickhouse is nullable, will insert null. If not, we will insert 0.
in doris/storrocks, if decimal is out of bounds or inf -inf nan, we insert null.

Intuitively this might work, but I am not sure if this will be too implicit for users. Did you check how flink connector handle these cases?

@lmatz
Copy link
Contributor

lmatz commented Apr 10, 2024

could you add tests for these edge cases in the integration tests? Thanks!

fix ci
@xxhZs xxhZs force-pushed the xxh/fix-clickhouse-sink-decimal branch from d8dd79c to 2ca5821 Compare April 10, 2024 07:15
@xxhZs xxhZs force-pushed the xxh/fix-clickhouse-sink-decimal branch 2 times, most recently from 4c80360 to 65fb97b Compare April 10, 2024 14:31
fix ci

fix

fix ci
@xxhZs xxhZs force-pushed the xxh/fix-clickhouse-sink-decimal branch from 65fb97b to 36a5fbb Compare April 11, 2024 03:06
@wenym1
Copy link
Contributor

wenym1 commented Apr 11, 2024

Due to the starrocks update, we removed some of the calibration logic for decimal

Is the change only compatible to the latest version of starrocks? Does it work on some previous versions of starrocks? If not we may add in our document and release note to notify users to be careful with the upgrade.

Copy link
Contributor

@wenym1 wenym1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM.

Can you paste some links from starrocks and doris about the related changes in the PR to explain the reason for removing the decimal config map from starrocks and changing the decimal config map from (u8, u8) to u8 for doris?

@xxhZs
Copy link
Contributor Author

xxhZs commented Apr 11, 2024

Due to the starrocks update, we removed some of the calibration logic for decimal

Is the change only compatible to the latest version of starrocks? Does it work on some previous versions of starrocks? If not we may add in our document and release note to notify users to be careful with the upgrade.

test with 2.5.18 -> laster(The versions we support), No problem

@xxhZs xxhZs added this pull request to the merge queue Apr 11, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 11, 2024
@xxhZs xxhZs enabled auto-merge April 11, 2024 07:40
@xxhZs xxhZs added this pull request to the merge queue Apr 11, 2024
Merged via the queue into main with commit 9dfbe22 Apr 11, 2024
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants