conditionally create mz_system and mz_probe cluster replicas

[SangJunBak]

See commit messages for details

Motivation

• This PR adds a known-desirable feature.
  https://github.com/MaterializeInc/database-issues/issues/8954

Tips for reviewer

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[SangJunBak]

@jubrad Unable to run bin/orchestratord to test due to the error

  Compiling mz-ore v0.1.0 (/Users/sangjunbak/materialize/src/ore)
error[E0432]: unresolved import `chrono`
  --> src/ore/src/panic.rs:31:5
   |
31 | use chrono::Utc;
   |     ^^^^^^ use of undeclared crate or module `chrono`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `mz-ore` (lib) due to 1 previous error
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/mzimage.py", line 135, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/mzimage.py", line 37, in main
    deps.acquire()
  File "/Users/sangjunbak/materialize/misc/python/materialize/mzbuild.py", line 1080, in acquire
    prep = self._prepare_batch(deps_to_build)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangjunbak/materialize/misc/python/materialize/mzbuild.py", line 1051, in _prepare_batch
    pre_image_prep[cls] = pre_image.prepare_batch(instances)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangjunbak/materialize/misc/python/materialize/mzbuild.py", line 525, in prepare_batch
    spawn.runv(cargo_build, cwd=rd.root)
  File "/Users/sangjunbak/materialize/misc/python/materialize/spawn.py", line 75, in runv
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['env', 'CMAKE_SYSTEM_NAME=Linux', 'CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-unknown-linux-gnu-cc', 'CARGO_TARGET_DIR=/Users/sangjunbak/materialize/target-xcompile', 'TARGET_AR=aarch64-unknown-linux-gnu-ar', 'TARGET_CPP=aarch64-unknown-linux-gnu-cpp', 'TARGET_CC=aarch64-unknown-linux-gnu-cc', 'TARGET_CXX=aarch64-unknown-linux-gnu-c++', 'TARGET_CXXSTDLIB=static=stdc++', 'TARGET_LD=aarch64-unknown-linux-gnu-ld', 'TARGET_RANLIB=aarch64-unknown-linux-gnu-ranlib', 'RUSTFLAGS=--cfg=tokio_unstable -Clink-arg=-Wl,--compress-debug-sections=zlib -Clink-arg=-Wl,-O3 -Csymbol-mangling-version=v0 --cfg=tokio_unstable -L/opt/homebrew/Cellar/aarch64-unknown-linux-gnu/0.1.0/bin/../aarch64-unknown-linux-gnu/sysroot/lib -Clink-arg=-fuse-ld=lld -Clink-arg=-B/opt/homebrew/opt/lld/bin', 'cargo', 'build', '--target', 'aarch64-unknown-linux-gnu', '--bin', 'orchestratord', '--package=mz-orchestratord', '--release']' returned non-zero exit status 101.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/orchestratord.py", line 422, in <module>
    main()
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/orchestratord.py", line 76, in main
    args.func(args)
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/orchestratord.py", line 80, in run
    acquire(
  File "/Users/sangjunbak/materialize/misc/python/materialize/cli/orchestratord.py", line 356, in acquire
    subprocess.check_call(["bin/mzimage", "acquire", image, *args])
  File "/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/mzimage', 'acquire', 'orchestratord']' returned non-zero exit status 1.

. I'm guessing it's because I added adapter-types to sqllogictest's cargo.toml? Regardless, I don't think this should block the review

[jubrad]

I'm not sure about the changes in src/orchestratord/src/controller/materialize/environmentd.rs

and some of the comments need to be cleaned up.

[ParkMyCar]

Overall LGTM, would wait to merge until you get @jubrad's approval for orchestratord changes though!

[ParkMyCar]

nit: it feels like this would be better written as if-else statements? If you want to prevent all of the Ok wrapping you can do:

let cluster_config = if cluster_name == "foo" {
    &self.foo_cluster
} else {
    return ...
}

[SangJunBak]

sgtm!

[ParkMyCar]

While you're here, can you add a comment explaining why these have a default of 0? e.g. that they are ephemeral clusters we spin up only to scrape analytics or for support when debugging

[SangJunBak]

sgtm!

[ParkMyCar]

nit: I would maybe allow 0..=2 for these ranges, seems like it could maybe be useful in the future

[ParkMyCar]

Specifying these 5 fields all the time feels brittle, e.g. easy to mix up the config values for two different clusters. A fix might be to use newtypes, e.g.

pub struct SystemClusterReplicationFactor(usize);
pub struct CatalogServerClusterReplicationFactor(usize);
...

But that seems quite tedious too. If you feel inspired thinking about how we can make this more succinct might be nice, but definitely not blocking!

[SangJunBak]

Specifying these 5 fields all the time feels brittle

hmm my Rust knowledge is kinda capped here! Like the cluster_config variables ? Would that require a new BootstrapBuiltinClusterConfig struct per <builtin cluster>ClusterReplicationFactor(usize)? Or could you do something like:

... 
   builtin_analytics_cluster_config: BootstrapBuiltinClusterConfig {
                    size: replica_size.clone(),
                    replication_factor: AnalyticsClusterReplicationFactor (ANALYTICS_CLUSTER_DEFAULT_REPLICATION_FACTOR),
                },

If we had to make a new ...ClusterConfig struct per builtin cluster, i kinda feel like it makes each config less generic which might be bad? Might be helpful to go over this in person since I'm genuinely curious!

[ParkMyCar]

Missing a config for the catalog_server replication factor, maybe that's intentional? It seems like it would be nice to have parity for all system clusters here

[SangJunBak]

Per Justin:

I think we should only include these options where it wouldn't be a complete footgun to set it to a value other than 1.

And I kinda agree with him here!

[jubrad]

LGTM