Local dev tooling for Raft + `QueuedTransactor` refactor by zonotope · Pull Request #1414 · fluree/db

zonotope · 2026-07-01T18:00:45Z

Adds a Docker-based local deployment orchestrator (scripts/local/stack) and an HTTP load harness (scripts/local/load/) for exercising the Raft consensus path — bringing up a single-node monolithic or N-node Raft cluster, injecting faults, driving configurable read/write workloads, and correlating metrics with consensus events. Also lands one consensus-layer refactor: a helper (QueuedTransactor::enqueue_and_await) that consolidates the five-way scaffolding across Committer methods and locks four previously-per-method invariants in one place.

What's in this PR

1. `fluree-db-consensus`: extract `enqueue_and_await`

fluree-db-consensus/src/raft/queued_transactor.rs — net -56 LOC (+114 / -170).

Each of the five Committer methods (transact, revert, merge, rebase, push) was hand-rolling the same ~30-line pipeline: build the QueuedRequest envelope → serde_json::to_vec → content_store.put → derive canonical_body_cid → sample applied_at_millis → build QueueSubmission → submit_and_await. Extracting this into enqueue_and_await collapses the middle of each method to a single call and structurally enforces four invariants that were previously enforced by code proximity:

body_cid is computed from the same envelope bytes that produced request_cid (so the state machine's body-hash dedup stays honest).
applied_at_millis is sampled once per submission (not once per derived field, which could mis-dedup on a slow build).
idempotency_cache_key is always built from the canonical format_ledger_id(name, branch) form (not a hand-rolled format!).
retry_eligible mirrors idempotency_key.is_some() (a divergence here would either spam Raft with non-idempotent retries or fail to retry idempotent ones).

Per-op preprocessing (raw-txn upload resolution on transact, per-commit CAS uploads on push, merge-target resolution on merge) stays at the call site — only the envelope-to-outcome plumbing is shared. Call sites now read as straight-line code: build the envelope, call enqueue_and_await, match the outcome. No closures, no inverted control flow.

Behavior preserved:

All 345 tests under --features raft still pass.
All 5 queued_transactor::tests (status-code mapping for snapshot-installed / branch-reset / purged / dropped / poisoned) still pass.
single_node_round_trip integration tests (5) still pass.
cargo fmt --check and cargo clippy both clean.

2. `scripts/local/stack` — Docker-based deployment orchestrator

Single bash entry point at scripts/local/stack with subcommand dispatch. ~1080 lines.

Deployment modes:

--mode monolithic (default): one Fluree server, no consensus.
--mode raft --nodes N (default 3): full Raft cluster with automatic bootstrap via /cluster/initialize + /cluster/add-learner + /cluster/change-membership.

Storage modes:

--storage ephemeral (default): docker named volumes, wiped on down.
--storage persistent: bind-mount to ./data/, survives down --keep-data.

Raft-mode-specific: nodes share a single shared-data docker volume mounted at /var/lib/fluree/data, while raft log state stays per-node (fluree-N-raft at /var/lib/fluree/raft). This mirrors the production topology where a Raft cluster sits over shared object storage — followers must be able to read commit envelopes the leader-owned worker wrote. Volume layout is annotated in the generator with the rationale.

Commands:

Command	Both modes	Raft-only
`up [flags]`	✓
`down [--keep-data]`	✓
`status`	✓
`logs [N] [-f] [--tail LINES]`	✓
`load [args...]`	✓
`kill <N>` (SIGKILL)	✓
`restart <N>`	✓
`pause <N>` (SIGSTOP — unresponsive but alive)	✓
`unpause <N>` (SIGCONT)	✓
`partition <N>` (docker network disconnect — split-brain)		✓
`heal <N>` (docker network connect --alias)		✓

Configuration is picked up from env vars: DEFAULT_RAFT_NODES, PUBLIC_PORT_BASE, RAFT_PORT_BASE, FLUREE_LOCAL_STORAGE. All can also be overridden via up flags.

Mode is remembered in compose.generated.yml as a header comment (# Mode: raft, # Nodes: 3); every subsequent command reads it via read_mode and adapts (e.g. status skips the /cluster/status section in monolithic mode; partition / heal explicitly reject with "partition" only applies to --mode raft (current: monolithic)"). Future consensus modes (bft, paxos, etc.) can be added by adding a new --mode value and a per-mode emit_service branch — the command surface stays the same.

3. `scripts/local/load/` — HTTP load harness (new crate)

Standalone Rust crate at scripts/local/load/ (excluded from the workspace so it doesn't pollute cargo build / cargo test from root). ~2100 LOC across ~10 source files.

Same tool against either backend — the URL list is the only thing that changes:

Single-node: --addrs http://localhost:8091
Raft cluster: --addrs http://localhost:8091,http://localhost:8092,http://localhost:8093
Behind a load balancer: --addrs https://fluree-lb.internal

Workload shapes:

Workload	Purpose
`single-pound`	One `CreateLedger` at t=0, then transact-only. Baseline single-queue ceiling.
`create-only`	Pure `CreateLedger` stream. `Command::CreateLedger` apply throughput in isolation.
`transact-only`	Transact against `--seeded-ledger` names. Pre-seeded steady state.
`query-only`	Query against `--seeded-ledger` names. Local read path (no consensus), read availability during chaos.
`mixed-rw`	Interleaved reads + writes at configurable ratio (`--mixed-write-every`, default 5). Read/write concurrency on the same ledger.
`wide-fanout`	Creates N ledgers over the run; transacts to whichever have landed. Per-branch work queues, ownership recalc under failure.
`multitenant`	Continuous mix: 1 in N ops is a `CreateLedger`, rest transact. Multi-tenant onboarding, ledger-count scaling.

Idempotency modes (--idempotency-mode, orthogonal to workload):

Mode	Behavior	Exercises
`anonymous` (default)	No key sent	Raw consensus throughput baseline
`unique`	Fresh key per request	`CachingCommitter` records outcomes + state machine writes `ApplyRecord`s, but nothing dedups
`pooled`	Keys drawn round-robin from `--idempotency-pool-size` (default 100); body deterministic-per-slot	Cache-hit dedup path: first N ops populate the cache, subsequent ops hit it

Cluster-aware routing:

Round-robin across --addrs list.
On 503 leader-change, retry against the next address once.
Sustained failures (network-error / timeout / server-error) against a target trigger a --blacklist-window (default 5s) cool-off before it's eligible again.
4xx (client-error) doesn't trigger the cool-off — it's a request-side signal, not a target-health one.

Metrics:

HDR-histogram-backed p50 / p95 / p99 / p99.9 / max per op kind and aggregate.
Per-outcome-class counters: success, idempotency-hit, leader-change, overloaded, timeout, network-error, client-error, server-error.
Live one-line-per-second progress printer, final per-op summary + outcome breakdown + top-10-ledgers distribution.

--watch-cluster annotation:

Accepts a comma-separated list of raft-port URLs. On each poll, walks the list starting from the last-successful URL (sticky preference); on failure of the preferred URL, transparently falls over to the next one and prints a failover: now polling <url> line.
Annotates the latency stream when leader / term / voter-set changes.
On voter-set changes, computes locally — via a rendezvous-hash mirror of fluree-db-consensus/src/raft/ownership.rs — how many currently-known ledger main-branch owners would reassign ("14/47 known ledger main-branch owners reassigned"). Mirror is unit-tested for behavioral parity with the consensus crate.

Plumbed into the stack script as ./stack load [...] — auto-populates --addrs from the running compose file and builds the tool on first invocation.

How to try it

cd scripts/local

# Monolithic (fast, no consensus)
./stack up
./stack load --workload single-pound --duration 30s
./stack down

# 3-node Raft cluster
./stack up --mode raft --nodes 3
./stack status                                       # per-node + /cluster/status view
./stack load --workload wide-fanout --duration 60s \
    --watch-cluster http://localhost:9091,http://localhost:9092,http://localhost:9093

# In another shell — chaos
./stack kill 1                                       # SIGKILL leader; watch election
./stack partition 3                                  # split-brain a follower
./stack pause 2                                      # unresponsive but alive
./stack heal 3
./stack restart 1

# Idempotency cache-hit path
./stack load --workload transact-only --seeded-ledger <name-from-summary> \
    --idempotency-mode pooled --idempotency-pool-size 10 --duration 30s

./stack down

./stack help for the full command list; ./stack help <command> for per-command details; ./stack load --tool-help for the load tool's own --help.

What's intentionally not in this PR

Named honestly so they're on the follow-up radar:

stack verify mode (consistency assertions during / after chaos: one leader at all times, followers converge to bounded lag, receipts match applied state). The tool right now is a load-and-chaos harness; it's not yet a validator.
stack scenario <name> (deterministic chaos timelines via TOML for bug reproduction).
Snapshot / log-compaction verification. Exercised organically by long runs; not tested explicitly.
Runtime membership changes (add-learner / change-membership during load).
CreateBranch / Push / Revert / Merge / Rebase workloads. Load harness op vocabulary is CreateLedger + Transact + Query.
Query peer scaffolding in the topology.
X-Fluree-Min-T bounded-wait consistency mode on the load harness.
Baseline monolithic-vs-raft comparison (stack compare).
Storage-layer failure simulation (disk full, slow disk).

bplatz · 2026-07-02T03:32:42Z

+    }
+
+    fn next_wide_fanout(&self, idx: u64) -> Option<Op> {
+        let creates_so_far = self.ledgers.len() as u64;


Create cap counts landed ledgers (ledgers.len()), but the create decision fires on every idx multiple. Under create latency + concurrency, many creates dispatch before any lands → overshoots --target-ledger-count. Count dispatched creates via an atomic instead.

bplatz · 2026-07-02T03:32:43Z

+        // next target is conservative but useful: it stops a single
+        // election from sinking every request in flight.
+        let outcome = if matches!(outcome, Outcome::LeaderChange) {
+            self.record_outcome(idx, outcome);


First-attempt LeaderChange is recorded only into target health; only the retry outcome reaches metrics. A leader-change that succeeds on retry is booked as Success, so the per-outcome breakdown undercounts leader-change — the signal this harness exists to measure.

bplatz · 2026-07-02T03:32:44Z

+            IdempotencyMode::Anonymous | IdempotencyMode::Unique => idx,
+            IdempotencyMode::Pooled => idx % self.tuning.idempotency_pool_size.max(1),
+        };
+        let subject_id = format!("http://load.fluree/{}/s{}", self.run_id, body_seed);


Pooled key and body derive from global idx only, never the target ledger. In wide-fanout/multitenant the same (key, body) hits different ledgers → cross-ledger false dedup, or the "first pool_size ops per ledger fill the cache" claim breaks. Scope the slot/body by ledger.

bplatz · 2026-07-02T03:32:44Z

+    if s.len() <= max {
+        s.to_string()
+    } else {
+        format!("{}…", &s[..max - 1])


&s[..max - 1] is a byte slice — a user-supplied --seeded-ledger name >40 bytes with a multi-byte char on the boundary panics at end-of-run. Use char-safe truncation.

bplatz · 2026-07-02T03:32:55Z

+        if !counts {
+            return;
+        }
+        let prev = health.consecutive_failures.fetch_add(1, Ordering::Relaxed);


consecutive_failures is never reset after the blacklist window expires, so a recovered target re-blacklists on a single failure instead of threshold consecutive ones — asymmetric with the "consecutive" contract.

bplatz · 2026-07-02T03:32:56Z

+            // separates them.
+            let body = resp.text().await.unwrap_or_default();
+            let lower = body.to_ascii_lowercase();
+            if lower.contains("not the leader") || lower.contains("leader") {


contains("leader") is checked before the overload branch, so a 503 body mentioning both is misclassified as LeaderChange and retried. ("not the leader" is also already subsumed by "leader".)

bplatz · 2026-07-02T03:32:57Z

+        };
+        let op_kind = op.kind;
+        let ledger = op.ledger.clone();
+        issued.fetch_add(1, std::sync::atomic::Ordering::Relaxed);


issued is incremented before dispatch, and check-then-fetch_add across workers overshoots the cap by up to concurrency; a shutdown mid-select! can also count an op that never ran, so issued can exceed recorded total.

bplatz · 2026-07-02T03:32:58Z

+    for i in $(seq 1 "$node_count"); do
+        local url
+        url="$(node_host_public_url "$i")/health"
+        while ! curl -fsS -m 2 "$url" >/dev/null 2>&1; do


Readiness is checked only on the public /health listener, but bootstrap dials the raft listener (9090). A node whose public port is up before its raft listener/alias is ready can fail add-learner. Verify raft readiness too before bootstrap.

bplatz · 2026-07-02T03:32:58Z

+        # alongside node_id and blocking, not nested under `addrs`.
+        local add_body
+        add_body="{\"node_id\":$i,\"raft_addr\":\"$(node_peer_raft_url "$i")\",\"client_addr\":\"$(node_peer_client_url "$i")\",\"blocking\":true}"
+        curl -fsS -X POST -H 'Content-Type: application/json' \


No -m timeout and no surrounding deadline (unlike the /health loop). With "blocking":true, a not-yet-ready or hung raft peer makes up hang indefinitely with no diagnostic rather than erroring. Same for the initialize/change-membership calls.

bplatz · 2026-07-02T03:32:59Z

+    local promote_body
+    promote_body="{\"members\":[$members_json],\"retain\":false}"
+    curl -fsS -X POST -H 'Content-Type: application/json' \
+        -d "$promote_body" "$leader_raft_url/cluster/change-membership" >/dev/null


change-membership result is not verified — if the voter set fails to converge, up still reports success. A post-bootstrap /cluster/status sanity check would surface a broken quorum here rather than later.

bplatz · 2026-07-02T03:33:00Z

+            --tail)
+                [ $# -ge 2 ] || die "--tail requires an argument"
+                tail_lines="$2"; shift 2 ;;
+            [1-9]|[1-9][0-9])


Node-id pattern caps at 99. --nodes allows ≥100, so logs 100 falls through to *) and dies with "Unknown argument". Use a numeric check like the other node-indexed commands.

zonotope added 8 commits June 30, 2026 00:03

extract fn to enqueue and await requests

de72ea5

add docker-based local cluster management command

5dc6d1d

add docker-based local fluree stack commands

98ea1b5

add query only workload

18ff5f0

add idempotency mode to exercise caches

e915ace

add shared volume in raft mode

62b59f1

add multi-url polling

efba870

add multiple urls for watch-cluster

6ccc34c

zonotope requested review from aaj3f and bplatz July 1, 2026 18:00

bplatz reviewed Jul 2, 2026

View reviewed changes

bplatz approved these changes Jul 2, 2026

View reviewed changes

zonotope added 8 commits July 2, 2026 14:56

add query peers with shared storage

144816d

count dispatched creates instead of landed creates

d7c1eb8

scope cache hits for a slot-body pair by ledger

99afc68

count leader change events independently

c513cbb

reset consecutive failures to 0 when blacklisted

d90cce6

check overload markers first to avoid misclassification to leader change

bcc8906

use character count instead of byte length for truncation

0a00773

ensure that issued == recorded total after all workers exit

24f4a21

zonotope added 4 commits July 3, 2026 10:33

bootstrap only after both the health check and cluster status are up

6bcc444

add diagnostic when a raft peer is hung preventing up from succeeding

fa76af2

poll until the leader reports expected members after change-membership

4468b8d

accept any positive integer for node id instead of capping at 99

f7c4ac8

zonotope merged commit c6a7efd into main Jul 3, 2026
13 checks passed

zonotope deleted the feature/raft-dev-env branch July 3, 2026 15:37

Navigation Menu

Uh oh!

Local dev tooling for Raft + `QueuedTransactor` refactor - #1414

Local dev tooling for Raft + `QueuedTransactor` refactor#1414
zonotope merged 20 commits into
mainfrom
feature/raft-dev-env

zonotope commented Jul 1, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

bplatz Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zonotope commented Jul 1, 2026

What's in this PR

1. fluree-db-consensus: extract enqueue_and_await

2. scripts/local/stack — Docker-based deployment orchestrator

3. scripts/local/load/ — HTTP load harness (new crate)

How to try it

What's intentionally not in this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. `fluree-db-consensus`: extract `enqueue_and_await`

2. `scripts/local/stack` — Docker-based deployment orchestrator

3. `scripts/local/load/` — HTTP load harness (new crate)