Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sink): Support cassandra sink #11878

Merged
merged 9 commits into from
Sep 7, 2023
Merged

feat(sink): Support cassandra sink #11878

merged 9 commits into from
Sep 7, 2023

Conversation

xxhZs
Copy link
Contributor

@xxhZs xxhZs commented Aug 24, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

support cassandra sink. We can build cassandra sink like README

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has totally checked 3998 files.

Valid Invalid Ignored Fixed
1757 2 2239 0
Click to see the invalid file list
  • java/connector-node/risingwave-sink-cassandra/src/main/java/com/risingwave/connector/CassandraFactory.java
  • java/connector-node/risingwave-sink-cassandra/src/main/java/com/risingwave/connector/CassandraUtil.java

fmt
@xxhZs xxhZs force-pushed the xxh/cassandra-sink branch 2 times, most recently from 2d9f9a6 to 3232d75 Compare August 25, 2023 14:15
@xxhZs xxhZs marked this pull request as ready for review August 28, 2023 03:19
@codecov
Copy link

codecov bot commented Aug 28, 2023

Codecov Report

Merging #11878 (4db9182) into main (c49e49d) will increase coverage by 0.01%.
Report is 133 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #11878      +/-   ##
==========================================
+ Coverage   70.20%   70.21%   +0.01%     
==========================================
  Files        1375     1375              
  Lines      229075   229075              
==========================================
+ Hits       160813   160846      +33     
+ Misses      68262    68229      -33     
Flag Coverage Δ
rust 70.21% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
src/connector/src/sink/remote.rs 53.09% <ø> (ø)

... and 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xxhZs xxhZs force-pushed the xxh/cassandra-sink branch 2 times, most recently from bf2b4a7 to 2d87b7a Compare August 29, 2023 10:20
.asRuntimeException();
}
batchBuilder.addStatement(
bindUpdateInsertStatement(this.stmtMap.get("update"), old, row));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, cassandra is upsert by design. I think we can use insert statement instead of update statement here to simplify the codes. I am also okay of keeping the current implementation.

@lmatz
Copy link
Contributor

lmatz commented Sep 4, 2023

As one user requested sink to SycllaDB and SycllaDB claims to be "compatible with Apache Cassandra", could you add the same test for SycllaDB?

SycllaDB also offers a docker image:https://hub.docker.com/r/scylladb/scylla/

fmt

fmt

fmt

fmt

fmt

fmt
@hzxa21
Copy link
Collaborator

hzxa21 commented Sep 6, 2023

As one user requested sink to SycllaDB and SycllaDB claims to be "compatible with Apache Cassandra", could you add the same test for SycllaDB?

SycllaDB also offers a docker image:https://hub.docker.com/r/scylladb/scylla/

We have added demos for both Apache Cassandra and ScyllaDB. They both work now.

@hzxa21 hzxa21 added the user-facing-changes Contains changes that are visible to users label Sep 6, 2023
@hzxa21
Copy link
Collaborator

hzxa21 commented Sep 6, 2023

Also, this is actually a user-facing change. Can you update the PR description to include the release note?

@xxhZs xxhZs added this pull request to the merge queue Sep 7, 2023
Merged via the queue into main with commit 7d940cf Sep 7, 2023
36 checks passed
@xxhZs xxhZs deleted the xxh/cassandra-sink branch September 7, 2023 14:02
Comment on lines +23 to 33
- use: minio

# If you want to use aws-s3, configure AK and SK in env var and enable the following lines:
# - use: aws-s3
# bucket: test-bucket

# If you want to create CDC source table, uncomment the following line
# - use: connector-node
- use: connector-node

# if you want to enable etcd backend, uncomment the following lines.
# - use: etcd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert these changes.

Little-Wallace added a commit to Little-Wallace/risingwave that referenced this pull request Sep 8, 2023
commit e097adc
Author: StrikeW <wangsiyuanse@gmail.com>
Date:   Fri Sep 8 13:16:15 2023 +0800

    fix: mysql-sink integration test (risingwavelabs#12161)

commit d972225
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Fri Sep 8 12:16:06 2023 +0800

    chore(object store): change streaming write buffer size to 16MB for gcs (risingwavelabs#12150)

commit 8bd524b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Sep 8 11:36:49 2023 +0800

    chore(deps): Bump xxhash-rust from 0.8.6 to 0.8.7 (risingwavelabs#12159)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c924c37
Author: Yufan Song <33971064+yufansong@users.noreply.github.com>
Date:   Thu Sep 7 10:55:57 2023 -0700

    feat(stream): add multiple server url feature into nats, delete useless field (risingwavelabs#12138)

commit 471aa2b
Author: William Wen <44139337+wenym1@users.noreply.github.com>
Date:   Thu Sep 7 22:31:44 2023 +0800

    refactor(source): specify cdc generic type parameter for different cdc source (risingwavelabs#12109)

commit 7d940cf
Author: Xinhao Xu <84456268+xxhZs@users.noreply.github.com>
Date:   Thu Sep 7 21:22:06 2023 +0800

    feat(sink): Support cassandra sink (risingwavelabs#11878)

    Co-authored-by: Patrick Huang <hzxa21@hotmail.com>

commit cc3faf6
Author: Yuhao Su <31772373+yuhao-su@users.noreply.github.com>
Date:   Thu Sep 7 18:58:44 2023 +0800

    feat(watermark): handle watermark in project_set (risingwavelabs#12128)

commit 4bb7c66
Author: Li0k <yuli@singularity-data.com>
Date:   Thu Sep 7 18:44:17 2023 +0800

    feat(storage): reduce group deltas for l0 task (risingwavelabs#12085)

commit b2dc4eb
Author: Eric Fu <eric@singularity-data.com>
Date:   Thu Sep 7 18:33:01 2023 +0800

    refactor(metrics): unify storage_metric_level & streaming_metric_level to 'metrics_level' (risingwavelabs#12099)

commit 5f75f9b
Author: Li0k <yuli@singularity-data.com>
Date:   Thu Sep 7 17:52:33 2023 +0800

    feat(storage): sepearte intra picker (risingwavelabs#12147)

commit f1672f7
Author: Richard Chien <stdrc@outlook.com>
Date:   Thu Sep 7 16:26:22 2023 +0800

    fix(eowc): fix SortBuffer when there are multiple records having same timestamp (risingwavelabs#12146)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit 183a98b
Author: Xu <xzhseh@gmail.com>
Date:   Thu Sep 7 04:23:15 2023 -0400

    feat(expr): add `array_max` (risingwavelabs#12100)

    Co-authored-by: Runji Wang <wangrunji0408@163.com>

commit 893f636
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 08:06:35 2023 +0000

    chore(deps): Bump chrono from 0.4.26 to 0.4.29 (risingwavelabs#12136)

    Signed-off-by: dependabot[bot] <support@github.com>
    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Runji Wang <wangrunji0408@163.com>

commit 3383291
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 15:49:07 2023 +0800

    chore(deps): Bump regex-automata from 0.3.7 to 0.3.8 (risingwavelabs#12134)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 5a8866d
Author: Li0k <yuli@singularity-data.com>
Date:   Thu Sep 7 15:28:56 2023 +0800

    refactor(storage): seperate validator logic from picker (risingwavelabs#11984)

commit b578db6
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Thu Sep 7 14:35:14 2023 +0800

    fix(parser): more error reporting with locations (risingwavelabs#12129)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit 3ee8a87
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Thu Sep 7 14:09:08 2023 +0800

    refactor(meta): refine `pause`/`resume` response (risingwavelabs#12079)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit f16809d
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Thu Sep 7 13:58:38 2023 +0800

    fix(expr): parse `timestamptz` without seconds but with offset (risingwavelabs#12084)

    Co-authored-by: StrikeW <wangsiyuanse@gmail.com>

commit 01ce1bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 11:56:53 2023 +0800

    chore(deps): Bump num-bigint from 0.4.3 to 0.4.4 (risingwavelabs#12137)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit d79113e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 10:42:13 2023 +0800

    chore(deps): Bump sysinfo from 0.29.9 to 0.29.10 (risingwavelabs#12135)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit ff4ef3a
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Thu Sep 7 10:03:57 2023 +0800

    fix(ci): fix backwards compat with `ci-release` profile (risingwavelabs#12139)

commit 9c98d26
Author: Xu <xzhseh@gmail.com>
Date:   Wed Sep 6 14:31:10 2023 -0400

    feat(expr): add `array_min` (risingwavelabs#12071)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: Runji Wang <wangrunji0408@163.com>

commit 084daa8
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Wed Sep 6 23:07:24 2023 +0800

    test: Add nexmark test, kafka test for backwards compatibility testing (risingwavelabs#10413)

commit 3c19064
Author: Yufan Song <33971064+yufansong@users.noreply.github.com>
Date:   Wed Sep 6 07:07:40 2023 -0700

    feat(stream): Add recovery feature for nats source (risingwavelabs#12073)

    Signed-off-by: tabVersion <tabvision@bupt.icu>
    Co-authored-by: tabVersion <tabvision@bupt.icu>

commit 52192e6
Author: Croxx <mrcroxx@outlook.com>
Date:   Wed Sep 6 19:18:32 2023 +0800

    refactor: refactor file cache metrics build, bump foyer (risingwavelabs#12121)

    Signed-off-by: MrCroxx <mrcroxx@outlook.com>

commit da89875
Author: William Wen <44139337+wenym1@users.noreply.github.com>
Date:   Wed Sep 6 18:27:52 2023 +0800

    feat(log-store): refine log store trait and add update vnode bitmap in reader (risingwavelabs#11959)

commit e64a075
Author: William Wen <44139337+wenym1@users.noreply.github.com>
Date:   Wed Sep 6 18:21:39 2023 +0800

    refactor(meta): introduce meta store ref and avoid pass generic parameter (risingwavelabs#12114)

commit b3a4195
Author: Yufan Song <33971064+yufansong@users.noreply.github.com>
Date:   Tue Sep 5 23:57:14 2023 -0700

    fix(stream): remove useless stream in nats sink (risingwavelabs#12107)

commit 68087f3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 06:24:42 2023 +0000

    chore(deps): bump flate2 from 1.0.26 to 1.0.27 (risingwavelabs#12103)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Bugen Zhao <i@bugenzhao.com>

commit e3f3d47
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Wed Sep 6 14:07:14 2023 +0800

    fix: temporarily remove the assert checking for split migration (risingwavelabs#12117)

commit 61ef35b
Author: Li0k <yuli@singularity-data.com>
Date:   Wed Sep 6 14:04:14 2023 +0800

    chore(storage): remove compactor workload (risingwavelabs#12005)

commit a76e519
Author: Dylan <chenzl25@mail2.sysu.edu.cn>
Date:   Wed Sep 6 13:17:22 2023 +0800

    refactor(java_binding): extract java_binding to jni_core (risingwavelabs#12108)

    Co-authored-by: Eric Fu <eric@singularity-data.com>
    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 99ecd5d
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Wed Sep 6 12:48:47 2023 +0800

    fix(streaming): handle missing `cache_policy` field in `OverWindow` (risingwavelabs#12110)

commit 50792cc
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Wed Sep 6 12:15:17 2023 +0800

    refactor(compaction): remove meta client from compactor context (risingwavelabs#12020)

commit b5dd428
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:52 2023 +0800

    chore(deps): bump openssl-sys from 0.9.92 to 0.9.93 (risingwavelabs#12106)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 1c87421
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:49 2023 +0800

    chore(deps): bump walkdir from 2.3.3 to 2.4.0 (risingwavelabs#12101)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 8c0da22
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:46 2023 +0800

    chore(deps): bump deranged from 0.3.7 to 0.3.8 (risingwavelabs#12102)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9ccdbe3
Author: zwang28 <70626450+zwang28@users.noreply.github.com>
Date:   Tue Sep 5 21:26:31 2023 +0800

    chore(rpc): set message size to max for stream client (risingwavelabs#12082)

commit 4f50903
Author: Eric Fu <eric@singularity-data.com>
Date:   Tue Sep 5 20:42:25 2023 +0800

    refactor(metrics): support streaming metrics level (risingwavelabs#11994)

commit 74c96fc
Author: Richard Chien <stdrc@outlook.com>
Date:   Tue Sep 5 19:47:23 2023 +0800

    perf(over window): fix over window performance degradation after risingwavelabs#11576 (risingwavelabs#12093)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit fdaf30e
Author: Richard Chien <stdrc@outlook.com>
Date:   Tue Sep 5 18:59:16 2023 +0800

    chore: add over window executor cache metrics (risingwavelabs#12092)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit ec0027e
Author: William Wen <44139337+wenym1@users.noreply.github.com>
Date:   Tue Sep 5 18:22:38 2023 +0800

    refactor(source): implement the common split reader as a generic function (risingwavelabs#12059)

commit 5cf0351
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Tue Sep 5 18:09:27 2023 +0800

    feat: modify the scale command to support both horizontal and vertical scaling. (risingwavelabs#12087)

commit 69bde53
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Tue Sep 5 18:03:18 2023 +0800

    feat: HA for schema registry (risingwavelabs#11982)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 477bfd4
Author: StrikeW <wangsiyuanse@gmail.com>
Date:   Tue Sep 5 16:57:55 2023 +0800

    fix(jdbc-sink):  quote identifier to handle case-sensitive case (risingwavelabs#12078)

commit aea50f5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 07:29:58 2023 +0000

    chore(deps): bump async-recursion from 1.0.4 to 1.0.5 (risingwavelabs#12069)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 579405e
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Tue Sep 5 14:43:46 2023 +0800

    feat(grafana): report local barrier manager progress (risingwavelabs#12001)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit e4b9a08
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Tue Sep 5 14:39:58 2023 +0800

    refactor(streaming): better traces for development (risingwavelabs#12024)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit 65fc959
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 06:37:26 2023 +0000

    chore(deps): bump tracing-opentelemetry from 0.20.0 to 0.21.0 (risingwavelabs#12010)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Runji Wang <wangrunji0408@163.com>

commit aadc605
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 14:17:42 2023 +0800

    chore(deps): bump google-cloud-pubsub from 0.19.0 to 0.20.0 (risingwavelabs#12066)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit a5ef008
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Tue Sep 5 14:05:59 2023 +0800

    feat(storage): refactor `LocalStateStore::init` and `StateTable::init` (risingwavelabs#12050)

commit ff623ee
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 03:49:12 2023 +0000

    chore(deps): bump syn from 2.0.29 to 2.0.31 (risingwavelabs#12067)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f95d972
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 03:49:01 2023 +0000

    chore(deps): bump memchr from 2.6.2 to 2.6.3 (risingwavelabs#12068)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 3f5a6c8
Author: Tao Wu <wutao@singularity-data.com>
Date:   Tue Sep 5 11:23:49 2023 +0800

    feat: support information_schema.views (risingwavelabs#12045)

commit 999b9d2
Author: wu <f_dogs@protonmail.com>
Date:   Mon Sep 4 18:10:08 2023 +0800

    feat: auto rebuild source reader in executor (risingwavelabs#11993)

    Co-authored-by: Bohan Zhang <tabvision@bupt.icu>

commit fa8b5e3
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Mon Sep 4 17:10:38 2023 +0800

    feat: add `target_parallelism_per_worker` field for scaling (risingwavelabs#11945)

commit deac61e
Author: Croxx <mrcroxx@outlook.com>
Date:   Mon Sep 4 17:07:26 2023 +0800

    feat(hummock): refill cache before apply version delta (risingwavelabs#11950)

    Signed-off-by: MrCroxx <mrcroxx@outlook.com>

commit 2ea66b8
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Mon Sep 4 16:15:30 2023 +0800

    chore: refine error message on unhandled error in kinesis (risingwavelabs#12055)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit dcfbfb5
Author: August <pin@singularity-data.com>
Date:   Mon Sep 4 14:50:29 2023 +0800

    feat: support for migration by new parallel units (risingwavelabs#11892)

commit 0d4da3b
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Mon Sep 4 14:10:49 2023 +0800

    fix: reject kinesis source if the start up mode is not sequence number but sequence number is provided (risingwavelabs#12048)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Li0k pushed a commit that referenced this pull request Sep 15, 2023
Co-authored-by: Patrick Huang <hzxa21@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
user-facing-changes Contains changes that are visible to users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants