test: add Kafka consumer-group, add-partition tests (inline style) #16244

xxchan · 2024-04-11T01:49:09Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

To think about changes in this PR, we can assume we implement this in a step to step manner:

We refer to existing inline tests (e.g., e2e_test/source/cdc_inline/alter/postgres_alter.slt) to write such test for Kafka. Now we explicitly use environment variables to set kafka brokers. Before running tests (CI/local), we set env vars.
- In this step, we used js to write some more complex scripts.
We added kafka broker env vars to risedev-env, so that we don't need to set envvars locally (when we use risedev slt.
We added user-managed Kafka in risedev.yml, so that we don't need to set envvars in CI. Now everything is unified.

The spirit of this PR, and more details about what's changed ("Example: Integrate Kafka" section) are described in #12451 (comment)

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

e2e_test/source_new/kafka/test_1.slt

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan · 2024-04-12T18:27:05Z

e2e_test/source_inline/commands.toml

@@ -0,0 +1,108 @@
+# This file contains commands used by the tests.


The commands added here are actually not used in the slt file (although in theory they can). They are now mainly helper commands to run manually.

e.g., risedev rpk is similar to risedev psql. It sets RPK_BROKERS env var. So it's helpful for local testing.

Signed-off-by: xxchan <xxchan22f@gmail.com>

commit b3c8e4e Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:39:13 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit a575c3a Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:38:15 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit df9f905 Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:26:00 2024 +0800 update Signed-off-by: xxchan <xxchan22f@gmail.com> commit b8859ca Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:25:14 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit f001e31 Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:23:06 2024 +0800 revert risedev Signed-off-by: xxchan <xxchan22f@gmail.com> commit 33aa178 Merge: 530eb86 c98dfd5 Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:22:13 2024 +0800 Merge branch 'xxchan/latin-tyrannosaurus' into xxchan/source-test commit 530eb86 Merge: 0b6f74c cf221e3 Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:21:42 2024 +0800 Merge branch 'xxchan/latin-tyrannosaurus' into xxchan/source-test Signed-off-by: xxchan <xxchan22f@gmail.com> commit 0b6f74c Author: xxchan <xxchan22f@gmail.com> Date: Sat Apr 13 02:11:40 2024 +0800 sleep more? Signed-off-by: xxchan <xxchan22f@gmail.com> commit 47c2c61 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 22:30:09 2024 +0800 sleep more Signed-off-by: xxchan <xxchan22f@gmail.com> commit 1853adf Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 21:59:31 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit 8711cf5 Merge: f1c0185 4f53f89 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 17:52:38 2024 +0800 Merge remote-tracking branch 'origin/main' into xxchan/source-test Signed-off-by: xxchan <xxchan22f@gmail.com> commit f1c0185 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 17:52:12 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit e9e2dc4 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 16:49:20 2024 +0800 install rpk Signed-off-by: xxchan <xxchan22f@gmail.com> commit e2f2a8e Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 15:16:44 2024 +0800 f Signed-off-by: xxchan <xxchan22f@gmail.com> commit bae6495 Merge: 58de398 4ac029c Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 15:11:54 2024 +0800 Merge remote-tracking branch 'origin/main' into xxchan/source-test commit 58de398 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 12 15:11:49 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit 9ba9728 Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 17:45:24 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit d8a2489 Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 17:18:38 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit c4b4b16 Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 17:16:25 2024 +0800 fix Signed-off-by: xxchan <xxchan22f@gmail.com> commit 6dddbf3 Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 17:15:00 2024 +0800 rename new to inline Signed-off-by: xxchan <xxchan22f@gmail.com> commit cf50f5e Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 17:04:54 2024 +0800 bump Signed-off-by: xxchan <xxchan22f@gmail.com> commit 9ca9d55 Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 16:00:07 2024 +0800 support user-managed kafka in risedev Signed-off-by: xxchan <xxchan22f@gmail.com> commit d4d405d Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 15:27:20 2024 +0800 update Signed-off-by: xxchan <xxchan22f@gmail.com> commit 6ed6cfc Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 10:21:25 2024 +0800 update commit f7a3dd5 Merge: 8b6c482 254ad0c Author: xxchan <xxchan22f@gmail.com> Date: Thu Apr 11 09:45:50 2024 +0800 Merge remote-tracking branch 'origin/main' into xxchan/source-test commit 8b6c482 Author: xxchan <xxchan22f@gmail.com> Date: Tue Apr 9 14:12:37 2024 +0800 update commit f280603 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 23:14:30 2024 +0800 add new source tests commit 0e3ace1 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 23:13:05 2024 +0800 revert unrelated change commit a3f3409 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 17:58:48 2024 +0800 fix commit 0a2ded0 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 17:27:45 2024 +0800 fix commit 8daa64d Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 17:19:38 2024 +0800 debug commit 8628c1f Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 17:15:32 2024 +0800 fix commit c8bde20 Author: xxchan <xxchan22f@gmail.com> Date: Fri Apr 5 17:00:09 2024 +0800 ci: install risedev to ci image Signed-off-by: xxchan <xxchan22f@gmail.com>

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan · 2024-04-16T11:48:43Z

e2e_test/source_inline/kafka/consumer_group.slt

+
+sleep 2s
+
+# Verify that RisingWave's Kafka consumer works independently from the console consumers subscribing to the same group.


This test case is intended for #16111

xxchan · 2024-04-16T11:49:34Z

e2e_test/source_inline/kafka/add_partition.slt

+control substitution on
+
+system ok
+rpk topic create test_add_partition -p 3


This test case is for #15994 (incomplete yet, still need some more cases)

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan · 2024-04-16T16:05:44Z

e2e_test/source_inline/kafka/consumer_group.slt

+sleep 5s
+
+system ok
+./e2e_test/source_inline/kafka/consumer_group.mjs --mv mv list-members


using js here is just a personal choice. It doesn't matter much. Can also use sh, py...

Here a script is needed and plain bash command is not enough because we need to first query fragment id, and then filter consumer groups with that.

Can we automatically download/install zx for developers running tests locally, like through cargo make? Or, include this in the developer documentation.

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan · 2024-04-17T03:04:28Z

src/risedevtool/src/risedev_env.rs

            }
+            ServiceConfig::Kafka(c) => {
+                let brokers = format!("{}:{}", c.address, c.port);
+                writeln!(env, r#"RISEDEV_KAFKA_BOOTSTRAP_SERVERS="{brokers}""#,).unwrap();
+                writeln!(env, r#"RISEDEV_KAFKA_WITH_OPTIONS_COMMON="connector='kafka',properties.bootstrap.server='{brokers}'""#).unwrap();
+                writeln!(env, r#"RPK_BROKERS="{brokers}""#).unwrap();
+            }


This is a core change

xxchan · 2024-04-17T03:04:45Z

risedev.yml

+      - use: kafka
+        user-managed: true
+        address: message_queue
+        port: 29092


This is a core change

e2e_test/source_inline/README.md

src/risedevtool/src/task/ensure_stop_service.rs

BugenZhao · 2024-04-17T03:12:45Z

e2e_test/source_inline/kafka/consumer_group.slt

+sleep 5s
+
+system ok
+./e2e_test/source_inline/kafka/consumer_group.mjs --mv mv list-members


Can we automatically download/install zx for developers running tests locally, like through cargo make? Or, include this in the developer documentation.

BugenZhao · 2024-04-17T03:15:20Z

e2e_test/source_inline/kafka/consumer_group.slt

+
+# The lag for batch query's group is 0, and each MV parition's group is 2 (1 of 3 consumed).
+system ok
+./e2e_test/source_inline/kafka/consumer_group.mjs --mv mv list-lags


So this requires that current working directory is at the project root. Can we organize these utilities together and add them to the PATH somehow? 😄

So this requires that current working directory is at the project root. Can we organize these utilities together and add them to the PATH somehow? 😄

I also thought about this, but not very sure. It can indeed make the script look nicer. (There's also another benefit: we can make the scripts a "project", and do more powerful stuff like adding dependencies to it.)

risedev always runs in the project root. We assume the slt is run by risedev slt, not only for the path issue, but also env vars, so even if we changed it, we still need to use risedev slt.

As @stdrc mentioned test: "inline" style connector e2e tests #12451 (comment), (also like DocSlt) maybe orginaze everything together is clearer. Adding more indirection also increases burden to find the scripts. (first indirection is put command in script, second indirection is put the script somewhere else and in PATH). Although running the tests doesn't require multiple steps now, but maybe not good for reading/writing.

So I'm a little hesitant about this.

I guess: if the script is ad-hoc (only used for one test), it's better to put next to the test file. If it is general (and as we have more such scripts), it's better to put together. It's kind of like "built-in" commands for the test driver. 😄

For this PR, I prefer not to change it as the benefits haven't shown.

BugenZhao · 2024-04-17T03:16:28Z

e2e_test/source_inline/kafka/consumer_group.slt

+rpk topic create test_consumer_group -p 3
+
+system ok
+cat <<EOF | rpk topic produce test_consumer_group -f "%p %v\\n" -p 0


Can we also extract this pipeline into a single command?

Can you suggest an improvement? I feel the command is simple enough. Adding a wrapper just makes it less clear (unnecessary indirection)

For instance, if we are going to adopt the same "%p %v\n" format in several places...

We will also use other formats, especially "%k %v\n" I guess

e2e_test/source_inline/commands.toml

Signed-off-by: xxchan <xxchan22f@gmail.com>

BugenZhao

🥰

Signed-off-by: xxchan <xxchan22f@gmail.com>

github-actions bot added the component/test Test related issue. label Apr 11, 2024

xxchan commented Apr 11, 2024

View reviewed changes

e2e_test/source_new/kafka/test_1.slt Outdated Show resolved Hide resolved

BugenZhao mentioned this pull request Apr 11, 2024

ci: install risedev to ci image #16161

Merged

9 tasks

xxchan marked this pull request as ready for review April 11, 2024 09:09

xxchan changed the title ~~test: introduce new source test (Kafka)~~ test: introduce new inline e2e source test (Kafka) Apr 12, 2024

xxchan mentioned this pull request Apr 12, 2024

test: "inline" style connector e2e tests #12451

Closed

style: improve source test and risedev-env style

cf221e3

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan mentioned this pull request Apr 12, 2024

style: minor improvements for source test and risedev #16289

Merged

9 tasks

xxchan changed the base branch from main to xxchan/latin-tyrannosaurus April 12, 2024 18:18

fix

c98dfd5

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan commented Apr 12, 2024

View reviewed changes

xxchan added 2 commits April 13, 2024 03:22

fmt

e11de32

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan force-pushed the xxchan/source-test branch from b3c8e4e to 143d569 Compare April 12, 2024 19:28

xxchan force-pushed the xxchan/latin-tyrannosaurus branch from 0ab4742 to 6f1d622 Compare April 12, 2024 19:50

xxchan added 2 commits April 13, 2024 03:52

why? would sleep longer help?

9504bba

Signed-off-by: xxchan <xxchan22f@gmail.com>

debug

bc1cce2

Base automatically changed from xxchan/latin-tyrannosaurus to main April 15, 2024 16:04

xxchan added 4 commits April 16, 2024 10:47

Merge branch 'main' into xxchan/source-test

75f6088

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

6abc141

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

dcb3cd7

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

8f3d927

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan commented Apr 16, 2024

View reviewed changes

xxchan added 6 commits April 16, 2024 20:11

fixx

0a98016

Signed-off-by: xxchan <xxchan22f@gmail.com>

f

b92c94f

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

e731307

Signed-off-by: xxchan <xxchan22f@gmail.com>

simplify script

2bef8a7

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

d656678

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix

e5fb620

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan commented Apr 16, 2024

View reviewed changes

remove a bad test.....

fdf05fb

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan changed the title ~~test: introduce new inline e2e source test (Kafka)~~ test: add Kafka consumer-group, add-partition tests (inline style) Apr 17, 2024

xxchan requested review from tabVersion and BugenZhao April 17, 2024 02:38

xxchan commented Apr 17, 2024

View reviewed changes

BugenZhao reviewed Apr 17, 2024

View reviewed changes

xxchan added 8 commits April 17, 2024 11:25

Merge remote-tracking branch 'origin/main' into xxchan/source-test

f4701ba

Signed-off-by: xxchan <xxchan22f@gmail.com>

bump ci image version

95581a9

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix conflict

dba9762

Signed-off-by: xxchan <xxchan22f@gmail.com>

fix typo

374106f

Signed-off-by: xxchan <xxchan22f@gmail.com>

minor

39a95a3

Signed-off-by: xxchan <xxchan22f@gmail.com>

debug

181a06c

Signed-off-by: xxchan <xxchan22f@gmail.com>

bump slt

1636afa

Signed-off-by: xxchan <xxchan22f@gmail.com>

Merge remote-tracking branch 'origin/main' into xxchan/source-test

f0f6716

xxchan requested a review from a team as a code owner April 17, 2024 11:38

sleep longer

3757c4f

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan force-pushed the xxchan/source-test branch from e309ef7 to 3757c4f Compare April 17, 2024 15:44

BugenZhao approved these changes Apr 18, 2024

View reviewed changes

xxchan added 4 commits April 18, 2024 12:47

remove debug log

5454343

Signed-off-by: xxchan <xxchan22f@gmail.com>

Merge remote-tracking branch 'origin/main' into xxchan/source-test

ea096a7

update readme

8c4be28

Signed-off-by: xxchan <xxchan22f@gmail.com>

refine EnsureStopService

9ab5045

Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan enabled auto-merge April 18, 2024 05:13

xxchan added this pull request to the merge queue Apr 18, 2024

Merged via the queue into main with commit 4b5efa2 Apr 18, 2024
28 of 29 checks passed

xxchan deleted the xxchan/source-test branch April 18, 2024 06:06

xxchan mentioned this pull request Apr 19, 2024

test: add some test for split change #15994

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add Kafka consumer-group, add-partition tests (inline style) #16244

test: add Kafka consumer-group, add-partition tests (inline style) #16244

xxchan commented Apr 11, 2024 •

edited

Loading

xxchan Apr 12, 2024 •

edited

Loading

xxchan Apr 16, 2024

xxchan Apr 16, 2024 •

edited

Loading

xxchan Apr 16, 2024

BugenZhao Apr 17, 2024

xxchan Apr 17, 2024

xxchan Apr 17, 2024

BugenZhao Apr 17, 2024

BugenZhao Apr 17, 2024

xxchan Apr 17, 2024

xxchan Apr 17, 2024

BugenZhao Apr 17, 2024

xxchan Apr 17, 2024 •

edited

Loading

BugenZhao Apr 17, 2024

xxchan Apr 17, 2024

BugenZhao left a comment

		@@ -0,0 +1,108 @@
		# This file contains commands used by the tests.


		sleep 2s

		# Verify that RisingWave's Kafka consumer works independently from the console consumers subscribing to the same group.

test: add Kafka consumer-group, add-partition tests (inline style) #16244

test: add Kafka consumer-group, add-partition tests (inline style) #16244

Conversation

xxchan commented Apr 11, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

xxchan Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xxchan Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xxchan Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BugenZhao left a comment

Choose a reason for hiding this comment

xxchan commented Apr 11, 2024 •

edited

Loading

xxchan Apr 12, 2024 •

edited

Loading

xxchan Apr 16, 2024 •

edited

Loading

xxchan Apr 17, 2024 •

edited

Loading