Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Introduce alter parallelism #14240

Merged
merged 2 commits into from Dec 28, 2023
Merged

feat: Introduce alter parallelism #14240

merged 2 commits into from Dec 28, 2023

Conversation

shanicky
Copy link
Contributor

@shanicky shanicky commented Dec 27, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

second part of #13954

This PR introduces the syntax ALTER TABLE/MATERIALIZED VIEW/SINK table_name SET PARALLELISM = AUTO/1,2,3…, following which the PARALLELISM status of a table can be observed within the internal rw_table_fragments table.

Altering a streaming job to AUTO will expand its degree of parallelism to encompass all available units, whereas setting it to a fixed number will lock the job’s parallelism at that specific figure. Setting it to 0 is equivalent to AUTO.

With the enable_automatic_parallelism_control option activated in the meta configuration, the addition of new compute nodes will automatically trigger an adjustment in parallelism.

Streaming jobs in AUTO mode will automatically be scaled up to include new compute nodes, whereas jobs in fixed mode will undergo a rebalance while maintaining the set level of parallelism.

dev=> set streaming_parallelism = 3;
SET_VARIABLE
dev=> create table t(v int);
CREATE_TABLE
dev=> select fragment_id, parallelism from rw_fragments;
 fragment_id | parallelism
-------------+-------------
           1 |           3
           2 |           3
(2 rows)

dev=> alter table t set parallelism = 8;
ALTER_TABLE
dev=> select fragment_id, parallelism from rw_fragments;
 fragment_id | parallelism
-------------+-------------
           1 |           8
           2 |           8
(2 rows)

dev=> alter table t set parallelism = auto;
ALTER_TABLE

dev=> select fragment_id, parallelism from rw_fragments;
 fragment_id | parallelism
-------------+-------------
           1 |          12
           2 |          12
(2 rows)

For the table we talked about before, we’re gonna treat it as custom by default to make sure it fits in okay. If you need to, you can switch it over to 'auto' or 'fixed' yourself to get that automatic control for how many things run at once, but that's only if the switch is turned on.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Add alter parallelism syntax so that users can control the degree of parallelism for the targeted streaming job.

@shanicky shanicky changed the title feat: Introduct alter parallelism feat: Introduce alter parallelism Dec 27, 2023
@shanicky shanicky added user-facing-changes Contains changes that are visible to users experimental labels Dec 28, 2023
@shanicky shanicky changed the base branch from main to peng/new-schedule December 28, 2023 07:32
Base automatically changed from peng/new-schedule to main December 28, 2023 07:34
Signed-off-by: Shanicky Chen <peng@risingwave-labs.com>
Signed-off-by: Shanicky Chen <peng@risingwave-labs.com>
Copy link
Contributor

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please add some details for this syntax in Release Note section.

@shanicky shanicky added this pull request to the merge queue Dec 28, 2023
Merged via the queue into main with commit f19ae82 Dec 28, 2023
35 checks passed
@shanicky shanicky deleted the peng/new-alter branch December 28, 2023 08:20
@ShanlanLi ShanlanLi added this to the release-1.6 milestone Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experimental type/feature user-facing-changes Contains changes that are visible to users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants