feat(cmd_all): support `single_node` mode #14951

kwannoel · 2024-02-02T06:55:44Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

single_node mode is a new mode for users who wish to run risingwave in a single process.

Previously users would rely on etcd and minio to run such an instance, now we use file system and embedded sqlite to power the state store and meta stores.

For the configurations of single_node, we don't need to expose node_level options to the end-user. As such we create a new options struct with the essential options.

For the internals, we just:

Map these options to node level options.
Pass the options to standalone_mode.
Use standalone_mode to spawn all these nodes in a single process.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

This is considered a "pre-release".

In this PR, we release a new mode: single_node mode.

./risingwave will invoke this new mode by default.

Setup

You can run it simply with the following steps:

git clone git@github.com:risingwavelabs/risingwave.git
git checkout kwannoel/standalone-easy-defaults
cd risingwave
./risedev build-risingwave
./target/debug/risingwave

Trying it

In a separate terminal connect to risingwave:

 psql -h localhost -p 4566 -d dev -U root

Cleaning the cluster

The default data directory is in ~/.risingwave. You can rm that directory to purge associated data.

Configurations

> ./target/debug/risingwave single_node --help

Usage: risingwave single_node [OPTIONS]

Options:
      --prometheus-listener-addr <PROMETHEUS_LISTENER_ADDR>
          The address prometheus polls metrics from [env: RW_SINGLE_NODE_PROMETHEUS_LISTENER_ADDR=]
      --config-path <CONFIG_PATH>
          The path to the cluster configuration file [env: RW_SINGLE_NODE_CONFIG_PATH=]
      --store-directory <STORE_DIRECTORY>
          The store directory used by meta store and object store [env: RW_SINGLE_NODE_STORE_DIRECTORY=]
      --meta-addr <META_ADDR>
          The address of the meta node [env: RW_SINGLE_NODE_META_ADDR=]
      --compute-addr <COMPUTE_ADDR>
          The address of the compute node [env: RW_SINGLE_NODE_COMPUTE_ADDR=]
      --frontend-addr <FRONTEND_ADDR>
          The address of the frontend node [env: RW_SINGLE_NODE_FRONTEND_ADDR=]
      --compactor-addr <COMPACTOR_ADDR>
          The address of the compactor node [env: RW_SINGLE_NODE_COMPACTOR_ADDR=]
  -h, --help
          Print help
  -V, --version
          Print version

Hiding other features

Playground mode will be hidden.
Standalone mode will be hidden.

github-actions

license-eye has totally checked 4785 files.

Valid	Invalid	Ignored	Fixed
2081	1	2703	0

Click to see the invalid file list

src/cmd_all/src/single.rs

src/compute/src/lib.rs

src/cmd_all/src/bin/risingwave.rs

fuyufjh · 2024-02-02T08:56:09Z

src/cmd_all/src/single_node.rs

+
+pub fn make_single_node_sql_endpoint(store_directory: &String) -> String {
+    format!(
+        "sqlite://{}/meta_store/single_node.db?mode=rwc",


Suggested change

"sqlite://{}/meta_store/single_node.db?mode=rwc",

"sqlite://{}/meta_store.db?mode=rwc",

Hmm why this change? IMO this structure looks more consistent:

noelkwan@Noels-MacBook-Pro risingwave % tree ~/.risingwave/ /Users/noelkwan/.risingwave/ ├── meta_store │ └── single_node.db └── state_store └── hummock_001 ├── checkpoint │ └── 0 └── cluster_id └── 0

Than this:

noelkwan@Noels-MacBook-Pro risingwave % tree ~/.risingwave/ /Users/noelkwan/.risingwave/ ├── single_node.db └── state_store └── hummock_001 ├── checkpoint │ └── 0 └── cluster_id └── 0

I think Eric's idea is:

/Users/noelkwan/.risingwave/ ├── meta_store.db └── state_store └── hummock_001 ├── checkpoint │ └── 0 └── cluster_id └── 0

src/cmd_all/src/bin/risingwave.rs

BugenZhao · 2024-02-03T10:37:24Z

Will take a look next weekday. 🥺

BugenZhao

For now we have 3 single-process deployment modes:

playground: all config managed
single-node: some config managed
standalone: almost no config managed

Intuitively, they're different from each other and exist for some different demand. IMO, however, they could be understood as a single mode with different sets of command-line options being set to the default value.

For example, imagine that we have a single mode like this:

risingwave single
[--preset <profile>]
# fields that will be mapped, use default value if absent
[--meta-address ".."]
[--config-file ".."]
# Extra options to override those from the preset
[--meta-extra-opts ".."]
[--compute-extra-opts ".."]
[--frontend-extra-opts ".."]
[--compactor-extra-opts ".."]

Then...

playground is single --preset in-memory
single-node is single --preset local-disk
standalone is single without preset and specify the address and extra options manually.

In this approach there's not a clear boundary of 3 modes. If users want to tweak a few options for playground or single-node, they don't have to fall back to standalone. Just simply provide some extra-opts.

BTW, the map_opts seems to actually achieve the idea I proposed for improving the UX for standalone mode in #12142 (comment)

For example, if we have specified the listen address of the meta service, then we don't need to specify the meta address again for compute and frontend services. That's why I propose this issue: perhaps we need a (cluster-level ?) configuration file for this as a better replacement for raw command-line arguments.

It's available only for single-node under current design, but I still believe it's a good to have for standalone. With the unified single --preset approach, I guess all single deployment modes can benefit from this.

BugenZhao · 2024-02-05T03:39:40Z

src/cmd_all/src/single_node.rs

+}
+
+impl SingleNodeOpts {
+    fn default_frontend_opts() -> FrontendOpts {


IMO these methods could be painful to maintain. I'd suggest specifying the mapped fields only and leave other fields default with ..Default::default().

Hmm not sure how to expose clap's defaults. e.g. the default values specified by default_value.

I think I tried ..Default::default() for FrontendOpts previously, but it gives an error that the trait is not implemented for FrontendOpts.

Oops. Thought it's serde.

BugenZhao · 2024-02-05T03:47:33Z

src/cmd_all/src/standalone.rs

+#[command(
+    version,
+    about = "The Standalone mode allows users to start multiple services in one process, it exposes node-level options for each service",
+    hide = true


Since we are now hiding the entrance of standalone, it may be more appropriate to assign the name to single_node instead, as discussed. 🤣

Rename single_node -> standalone? Not quite sure what you mean.

We hide it from the end user, but cloud will still invoke it with ./risingwave standalone.

kwannoel · 2024-02-05T04:47:56Z

For now we have 3 single-process deployment modes:
* `playground`: all config managed

* `single-node`: some config managed

* `standalone`: almost no config managed
Intuitively, they're different from each other and exist for some different demand. IMO, however, they could be understood as a single mode with different sets of command-line options being set to the default value.

For example, imagine that we have a single mode like this:
risingwave single
[--preset <profile>]
# fields that will be mapped, use default value if absent
[--meta-address ".."]
[--config-file ".."]
# Extra options to override those from the preset
[--meta-extra-opts ".."]
[--compute-extra-opts ".."]
[--frontend-extra-opts ".."]
[--compactor-extra-opts ".."]
Then...
* `playground` is `single --preset in-memory`

* `single-node` is `single --preset local-disk`

* `standalone` is `single` without `preset` and specify the address and extra options manually.
In this approach there's not a clear boundary of 3 modes. If users want to tweak a few options for playground or single-node, they don't have to fall back to standalone. Just simply provide some extra-opts.

BTW, the map_opts seems to actually achieve the idea I proposed for improving the UX for standalone mode in #12142 (comment)

For example, if we have specified the listen address of the meta service, then we don't need to specify the meta address again for compute and frontend services. That's why I propose this issue: perhaps we need a (cluster-level ?) configuration file for this as a better replacement for raw command-line arguments.

It's available only for single-node under current design, but I still believe it's a good to have for standalone. With the unified single --preset approach, I guess all single deployment modes can benefit from this.

Some considerations:

I think that the default, if no preset is provided, should be to default to single-node. Otherwise having to specify the preset option is troublesome for users. It must be as simple as ./risingwave, if they want to run it.
For default options, their defaults will all be determined by what makes sense for single-node mode.

Otherwise, seems like this can unify all 3 execution modes and reduce maintenance overhead.
Let me give it a try.

kwannoel · 2024-02-05T07:09:17Z

Otherwise, seems like this can unify all 3 execution modes and reduce maintenance overhead.
Let me give it a try.

Some extra work needs to be done:

Need to provide some proc macro to generate the extra override opts from existing options.

--meta-extra-opts="
  --prometheus-addr="localhost:123"
  > default options
"


#[derive(Debug, Clone, Parser, OverrideConfig)]
#[command(version, about = "The central metadata management service")]
pub struct MetaNodeOpts {
    #[clap(long, env = "RW_VPC_ID")]
    vpc_id: Option<String>,

    #[clap(long, env = "RW_VPC_SECURITY_GROUP_ID")]
    security_group_id: Option<String>,

    // TODO: use `SocketAddr`
    #[clap(long, env = "RW_LISTEN_ADDR", default_value = "127.0.0.1:5690")]
    pub listen_addr: String,

Proc macro output:

pub struct MetaNodeOptsBuilder {
    #[clap(long, env = "RW_VPC_ID")]
    vpc_id: Option<Option<String>>,

    #[clap(long, env = "RW_VPC_SECURITY_GROUP_ID")]
    security_group_id: Option<Option<String>>,

    // TODO: use `SocketAddr`
    #[clap(long, env = "RW_LISTEN_ADDR", default_value = "127.0.0.1:5690")]
    pub listen_addr: Option<String>,
}

Thanks @BugenZhao for mentioning:
Use update_matches to pass in string options, no need to maintain a builder it seems:
https://docs.rs/clap/latest/clap/trait.Parser.html#method.update_from.

kwannoel · 2024-02-05T09:17:00Z

Discussed offline with bugen, we can merge this PR, and update the internals later to unify standalone, playground and single_node.

Track: #14997

Co-authored-by: Noel Kwan <47273164+kwannoel@users.noreply.github.com> Co-authored-by: Noel Kwan <noelkwan1998@gmail.com>

kwannoel added 9 commits February 2, 2024 14:34

add hack to support risingwave as entrypoint

3940e2c

add single_node interface + skeleton file

8901a4b

refactor away parse logic

803d90e

interim commit add data directory

fab3887

add standalone defaults

6250c19

add sql endpoint defaults

cb500b4

register state store, meta store

b952d8d

fix several bugs for unsupplied parameters

f8d4af7

fix conflict change + add --backend option

d84d076

kwannoel added the user-facing-changes Contains changes that are visible to users label Feb 2, 2024

kwannoel requested a review from a team as a code owner February 2, 2024 06:55

github-actions bot added the type/feature label Feb 2, 2024

kwannoel mentioned this pull request Feb 2, 2024

Add RisingWave single_node mode #14895

Open

22 tasks

github-actions bot reviewed Feb 2, 2024

View reviewed changes

kwannoel force-pushed the kwannoel/standalone-easy-defaults branch from 027206e to d84d076 Compare February 2, 2024 07:01

rm empty

4f77db0

xxchan reviewed Feb 2, 2024

View reviewed changes

src/compute/src/lib.rs Outdated Show resolved Hide resolved

xxchan reviewed Feb 2, 2024

View reviewed changes

src/cmd_all/src/bin/risingwave.rs Show resolved Hide resolved

allow config store_directory

a57cf07

kwannoel force-pushed the kwannoel/standalone-easy-defaults branch from 6560863 to a57cf07 Compare February 2, 2024 07:49

kwannoel added 2 commits February 2, 2024 16:18

move single node defaults configs into its module

8b38319

add some aliases

4b75db3

kwannoel added the need-cherry-pick-release-1.7 label Feb 2, 2024

kwannoel added 2 commits February 2, 2024 16:32

docs

2340acb

docs

76f1c08

kwannoel requested review from lmatz, fuyufjh, BugenZhao and xxchan February 2, 2024 08:39

stdrc self-requested a review February 2, 2024 09:01

fuyufjh approved these changes Feb 2, 2024

View reviewed changes

hide standalone, update docs for playground

d2aef0f

kwannoel mentioned this pull request Feb 5, 2024

Build and publish RisingWave binaries for linux arm64 #14898

Closed

Merge branch 'main' into kwannoel/standalone-easy-defaults

3e4885a

BugenZhao reviewed Feb 5, 2024

View reviewed changes

kwannoel added this pull request to the merge queue Feb 5, 2024

kwannoel mentioned this pull request Feb 5, 2024

Refactor: reuse single_node mode for playground and standalone #14997

Closed

Merged via the queue into main with commit afc3028 Feb 5, 2024
27 of 28 checks passed

kwannoel deleted the kwannoel/standalone-easy-defaults branch February 5, 2024 09:52

neverchanje mentioned this pull request Feb 5, 2024

Document: feat(cmd_all): support single_node mode risingwavelabs/risingwave-docs#1843

Closed

github-actions bot pushed a commit that referenced this pull request Feb 5, 2024

feat(cmd_all): support single_node mode (#14951)

dda4a53

github-actions bot mentioned this pull request Feb 5, 2024

feat(cmd_all): support single_node mode (#14951) #15006

Merged

kwannoel added a commit that referenced this pull request Feb 5, 2024

feat(cmd_all): support single_node mode (#14951)

896881d

kwannoel added a commit that referenced this pull request Feb 5, 2024

feat(cmd_all): support single_node mode (#14951) (#15006)

392a6ab

Co-authored-by: Noel Kwan <47273164+kwannoel@users.noreply.github.com> Co-authored-by: Noel Kwan <noelkwan1998@gmail.com>

WanYixian mentioned this pull request Feb 27, 2024

Doc update for single-node mode risingwavelabs/risingwave-docs#1882

Merged

2 tasks

xxchan mentioned this pull request Mar 11, 2024

feat(standalone): add extra opts to single node #15577

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cmd_all): support `single_node` mode #14951

feat(cmd_all): support `single_node` mode #14951

kwannoel commented Feb 2, 2024 •

edited

Loading

github-actions bot left a comment

fuyufjh Feb 2, 2024

kwannoel Feb 2, 2024

kwannoel Feb 2, 2024

neverchanje Feb 8, 2024

BugenZhao commented Feb 3, 2024

BugenZhao left a comment •

edited

Loading

BugenZhao Feb 5, 2024

kwannoel Feb 5, 2024

BugenZhao Feb 5, 2024

BugenZhao Feb 5, 2024

kwannoel Feb 5, 2024

kwannoel commented Feb 5, 2024

kwannoel commented Feb 5, 2024 •

edited

Loading

kwannoel commented Feb 5, 2024 •

edited

Loading

	"sqlite://{}/meta_store/single_node.db?mode=rwc",
	"sqlite://{}/meta_store.db?mode=rwc",

feat(cmd_all): support single_node mode #14951

feat(cmd_all): support single_node mode #14951

Conversation

kwannoel commented Feb 2, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

Setup

Trying it

Cleaning the cluster

Configurations

Hiding other features

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BugenZhao commented Feb 3, 2024

BugenZhao left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwannoel commented Feb 5, 2024

kwannoel commented Feb 5, 2024 • edited Loading

kwannoel commented Feb 5, 2024 • edited Loading

feat(cmd_all): support `single_node` mode #14951

feat(cmd_all): support `single_node` mode #14951

kwannoel commented Feb 2, 2024 •

edited

Loading

BugenZhao left a comment •

edited

Loading

kwannoel commented Feb 5, 2024 •

edited

Loading

kwannoel commented Feb 5, 2024 •

edited

Loading