[Phase 4] Cluster config retrieval from GCS bucket (GCP) and Consul KV (local)

[cbaugus]

After the Raft cluster forms and a leader is elected, the leader retrieves the test configuration from a centralized remote source rather than requiring identical env vars or YAML files on every node. The leader then distributes the parsed config to all followers via gRPC (Issue #46).

Cluster mode must be enabled

This feature only applies when CLUSTER_ENABLED=true. In standalone mode (the default) config continues to come from env vars / YAML files — no change to existing behaviour.

Config sources

GCS Bucket — GCP deployments

Leader downloads a YAML config object from GCS after election:

• Uses GCP Application Default Credentials (ADC) — no extra auth config needed on GCP VMs or container-as-a-service; the metadata server provides credentials automatically
• Config object is the same YAML schema already supported by the config system
• google-cloud-storage Rust crate (or reqwest against the GCS JSON API) for download

CLUSTER_CONFIG_SOURCE=gcs
GCS_CONFIG_BUCKET=my-loadtest-configs
GCS_CONFIG_OBJECT=configs/prod-test.yaml

Consul KV — local / dev deployments

Leader reads config from Consul KV store after election. The Consul agent is already running for service discovery and DNS (Issue #47) — the KV lookup uses the same agent, no additional connection needed.

• Key value must be a YAML string matching the existing config schema
• Consul blocking queries can be used for optional hot-reload when the KV value is updated while a test is running
• The leader's Consul health tag transitions to leader (Issue [Phase 4] Leader election and coordination (Raft) #47) before the KV fetch, so operators can verify the right node is fetching config

CLUSTER_CONFIG_SOURCE=consul-kv
CONSUL_CONFIG_KEY=loadtest/config   # KV path (default: loadtest/config)

Write config to Consul KV before starting a test:

consul kv put loadtest/config @my-test-config.yaml

Flow

1. CLUSTER_ENABLED=true → nodes start, discover peers
   - Consul:  query loadtest-cluster.service.consul for peers
   - Static:  read CLUSTER_NODES list
2. Raft cluster forms, leader elected (Issue #47)
3. Leader updates Consul tag to leader and /health/cluster returns state=leader
4. Leader reads CLUSTER_CONFIG_SOURCE env var
5. Leader fetches config:
   - gcs:       GCS download via ADC
   - consul-kv: Consul KV GET at CONSUL_CONFIG_KEY
6. Leader parses YAML → existing Config struct (no new schema needed)
7. Leader pushes TestConfig proto to all followers via gRPC DistributeConfig (Issue #46)
8. Followers acknowledge → health check returns cluster_ready=true on all nodes
9. Leader issues StartTest with coordinated start_at timestamp (Issue #48)

Error handling

• GCS object not found or ADC credentials fail → leader logs error and aborts (config problem, not a leader problem — re-election will not help)
• Consul KV key missing → abort with clear error: consul kv put loadtest/config @config.yaml
• Followers that do not receive config within CLUSTER_CONFIG_TIMEOUT_SECS (default 30s) log a warning; leader retries delivery up to 3 times before aborting
• Config parse error → abort with line/field details from the YAML parser

Standalone mode (CLUSTER_ENABLED=false — the default)

No change. Config comes from env vars and/or local YAML file as it does today. GCS and Consul KV are not consulted.

Full env var reference

CLUSTER_CONFIG_SOURCE=gcs          # gcs | consul-kv (required when CLUSTER_ENABLED=true)
GCS_CONFIG_BUCKET=                 # GCS bucket name (gcs source only)
GCS_CONFIG_OBJECT=                 # GCS object path e.g. configs/test.yaml (gcs source only)
CONSUL_CONFIG_KEY=loadtest/config  # Consul KV path (consul-kv source only)
CLUSTER_CONFIG_TIMEOUT_SECS=30     # max seconds to wait for config distribution to followers

[cbaugus]

Implementation complete (PR on dev branch)

Files changed

File	Change
Cargo.toml	Added base64 = "0.22"
src/lib.rs	Registered pub mod config_source;
src/config_source.rs	New — ConfigSource enum, from_env(), fetch(), GCS + Consul KV backends, percent_encode(), unit tests
src/main.rs	Leader auto-fetch task spawned after the config-watcher block; new env vars added to --help output
nomad/loadtest.nomad.hcl	Commented-out example blocks for both consul-kv and gcs sources
docs/CLUSTER_CONFIG_SOURCE.md	New — full operator guide covering Consul KV and GCS setup, IAM, Nomad template snippet, observability

---

Implementation notes

ConfigSource::from_env() returns None when CLUSTER_CONFIG_SOURCE is unset — auto-fetch is opt-in and has no effect on standalone or manually-configured clusters.

GCS fetch uses the GCE metadata server for ADC tokens — no SDK, no credentials file, works automatically on GCE VMs, GKE, Cloud Run, and Compute Engine-backed Nomad clients. Object paths with slashes are percent-encoded by percent_encode() so configs/prod.yaml resolves correctly.

Consul KV fetch calls GET /v1/kv/{key} against CONSUL_ADDR (default http://127.0.0.1:8500) and base64-decodes the Value field from the response array.

Leader detection watches raft.metrics() (openraft's built-in watch::Receiver<RaftMetrics>) for transitions where current_leader == this_node_id && !was_leader. Fetch fires exactly once per leader election.

CLUSTER_CONFIG_TIMEOUT_SECS (default 30) wraps the entire fetch in tokio::time::timeout — a hung metadata call or slow GCS download will not block the leader indefinitely.

Version string is auto-<unix-timestamp> so each auto-fetched config gets a distinct version visible in logs and metrics.

---

Quick-start (Consul KV)

# Write config to Consul before deploying the cluster
consul kv put loadtest/config @my-test-config.yaml

# Deploy with auto-fetch enabled
CLUSTER_ENABLED=true \
CLUSTER_CONFIG_SOURCE=consul-kv \
CONSUL_CONFIG_KEY=loadtest/config \
DISCOVERY_MODE=consul \
./rust_loadtest

Expected leader log output:

INFO Leader committed auto-fetched config (Issue #76)

---

What is deferred

• Consul blocking-query hot-reload (optional per spec)
• Periodic GCS polling
• Re-fetch after a node steps down and is re-elected (can be added by resetting was_leader on step-down)

[cbaugus]

Closing — fully implemented and verified in production

What shipped

Component	Details
src/config_source.rs	ConfigSource enum with GCS and ConsulKv variants; from_env() returns None when auto-fetch is disabled; fetch() pulls YAML from GCS (via GCE metadata ADC) or Consul KV (base64-decode)
src/main.rs	Leader auto-fetch task watches raft.metrics() for leader transitions and commits the fetched YAML to the Raft log on first election
src/config.rs	Config::from_yaml() — YAML-authoritative parser for Raft config apply; YAML values win over startup env-var defaults (NUM_CONCURRENT_TASKS, TARGET_RPS, TARGET_URL, etc.)

Bug fixed (discovered during production verification)

Config::from_yaml_with_env_overrides() used env > YAML precedence, so startup defaults set in the Nomad job (NUM_CONCURRENT_TASKS=25, TARGET_RPS=200) permanently shadowed the YAML values pushed via POST /cluster/config or auto-fetched from Consul KV. Workers stayed at 25 and RPS at 200 regardless of what the YAML said.

Fixed by adding Config::from_yaml() (YAML > env) and switching the Raft config-watcher to use it. Verified in production: pushing workers: 75 / target: 25000 now applies correctly across the 3-node cluster.

Verified in production (3-node Nomad cluster)

INFO Proxying config submission to Raft leader leader=192.168.2.22:7000 version=auto
INFO Applied test config from Raft log version=auto
INFO Raft config committed — draining worker pool workers=75 url=http://dialtone.service.consul:5678 load_model=Rps { target_rps: 25000.0 }
INFO Leader committed auto-fetched config (Issue #76)