Repository navigation
Example Dapr orders
A small Python service that must apply each order exactly once, tested against a simulated Dapr sidecar that injects duplicate deliveries, reordering and app crashes from a seed. It is the easiest of the process-style examples to try, because the simulator can run your app directly, with no Hermit, on a Mac or on Linux.
Files: examples/dapr-orders. Background: the Dapr section of the README and docs/determinism.md.
See a crash-recovery bug appear for some seeds and not others, see the fixed consumer survive the same faults, and understand how the same command is used as a process target under Hermit.
buggy_app.py deduplicates redelivered messages with an in-memory set. That works until the app restarts: the set is lost, the broker delivers the order again (Dapr pub/sub is at-least-once), and the order is applied twice. The Dapr state store survives the restart; the app's memory does not.
fixed_app.py deduplicates with a first-write-wins marker in the state store, so it survives restarts. Everything else is identical.
| Need | For |
|---|---|
| A Rust toolchain | building the simulator, hairspring-dapr-sim. It is not part of the release binaries |
| Python 3 | the app (standard library only) |
| A clone of the repository | the example files |
| For the Hermit part only: Linux x86-64 and Hermit (Installation) | exploring seeds under Hermit |
cargo build --release -p hairspring-daprThis puts target/release/hairspring-dapr-sim in the clone. It took a few seconds on our Mac once the workspace had been built. Its only documented option set is:
hairspring-dapr-sim: usage: hairspring-dapr-sim [--scenario FILE] [--seed N] [-- APP COMMAND...]
scenario.toml is the workload and fault plan:
[app]
env = { EXPECTED_IDS = "1,2,3", PYTHONDONTWRITEBYTECODE = "1", PYTHONPYCACHEPREFIX = "/hairspring-empty-pycache", PYTHONHASHSEED = "0" }
[faults]
duplicate = 0.5
reorder = 0.4
crash_app = 0.3
[[publish]]
topic = "orders"
data = { id = 1 }
# ... and orders 2 and 3Three orders are published. For each delivery, the simulator draws from a seeded random number generator: with probability 0.5 it redelivers a message that was already processed, with 0.4 it delivers out of order, and with 0.3 it kills and restarts the app after a delivery, losing its memory. When the run ends the simulator asks the app to check itself (GET /hairspring/check); the app answers 500 with order N was applied M times if any order was not applied exactly once. The simulator turns that into a line starting HAIRSPRING_VIOLATION:.
The other fault types the simulator supports are drop, state_conflict and state_error. Only an implemented subset of Dapr is simulated: state with ETags and first-write concurrency, publish, and subscription delivery. No bindings, secrets, actors, workflows or gRPC.
From examples/dapr-orders:
../../target/release/hairspring-dapr-sim --scenario scenario.toml --seed 1 -- python3 buggy_app.py
echo $?Expected output and exit status (we ran this):
HAIRSPRING_VIOLATION: order 1 was applied 2 times
and exit status 1. Now a seed that passes:
../../target/release/hairspring-dapr-sim --scenario scenario.toml --seed 2 -- python3 buggy_app.pyhairspring-dapr-sim: ok (seed 2): 4 delivered, 1 redelivered, 0 crashes
with exit status 0. The repository's README suggests seed 3 for the first command. On the Mac we used, the buggy app passes at seed 3, so use seed 1 to see the failure. (The README says "run it directly (no Hermit, so faults are seeded but thread timing is not)", which is the point: without Hermit the faults repeat for a seed but the app's own thread timing does not.)
We ran seeds 1 to 12 against the buggy app, and 1 to 8 against the fixed one:
| Seed | buggy_app.py |
fixed_app.py |
|---|---|---|
| 1 | violation: order 1 applied 2 times | ok (2 crashes) |
| 2 | ok | ok |
| 3 | ok | ok |
| 4 | violation: order 1 applied 2 times | ok (3 crashes) |
| 5 | violation: order 2 applied 2 times | ok (2 crashes) |
| 6 | violation: order 3 applied 2 times | ok (2 crashes) |
| 7 | ok | ok (1 crash) |
| 8 | ok | ok |
| 9, 10 | violation | not run |
| 11, 12 | ok | not run |
Run the same seed twice and you get the same answer (we repeated seeds 1 and 4 and saw the same output). That is the sidecar's fault plan being seeded. It does not show that the app's thread timing is repeatable without Hermit.
To run the fixed app: replace buggy_app.py with fixed_app.py in the command.
For a buggy_app.py seed that failed, the line names the broken invariant: which order, applied how many times. In the passing line, 4 delivered, 1 redelivered, 2 crashes is the fault plan's work for that seed: the number of delivery attempts, how many were redeliveries, and how many times the app was killed. A passing seed with 0 crashes says little, because the bug needs a crash between a delivery and its redelivery. A seed that fails is more informative than a seed that passes.
Compare the two apps on a seed where the fixed app saw crashes (seeds 1, 4 and 5): it survived several restarts and the order was still applied once.
Everything in this step comes from the example's README. We could not run it on a Mac. In hairspring.toml:
[target.orders-buggy]
kind = "process"
cwd = "/path/to/examples/dapr-orders"
command = ["/path/to/hairspring-dapr-sim", "--scenario", "scenario.toml", "--", "/usr/bin/python3", "buggy_app.py"]
[target.orders-buggy.budget]
max_trials = 40then hairspring test orders-buggy. The Hermit backend sets HAIRSPRING_SEED to each trial's seed, and the simulator uses it to draw its faults. So the seed now controls both the faults and Hermit's thread scheduling. Add hairspring test orders-buggy --sweep --trials 40 for a rate, and make a second target for fixed_app.py to compare.
The project's determinism notes record, from earlier testing on a Linux machine, that the buggy consumer failed with order 1 was applied 2 times at sched-seed 1 and repeated at that seed 3 of 3 times, and that the fixed consumer passed 12 of 12 seeds. We did not repeat that run. Treat it as a record of one setup, not a rate.
What happens on a Mac if you try step 6 anyway (we ran this):
hairspring: exploring orders-buggy (seed 1)
ERROR orders-buggy: hermit executable not found at `hermit`. Hermit runs only on x86-64 Linux: install it there, or set HAIRSPRING_HERMIT to its path
The command exits with status 2. See Troubleshooting.
-
buggy_app.py: theseen = set() # BUG: lost on restartline, andfirst_time(). -
fixed_app.py:first_time()writing aseen-<id>marker with"options": {"concurrency": "first-write"}, and returning true (first time seen) only when the state store answers 204. -
scenario.toml: thePYTHON*variables freeze Python's bytecode caches. They are there because Python rewrites.pycfiles, which changes what runs next time; see "Hidden inputs" in docs/determinism.md.
Nothing is installed. The simulator writes nothing into the example directory (we checked git status). Remove the build output with cargo clean -p hairspring-dapr if you wish.
The Dapr section of the README, examples/dapr-orders, and the hairspring-dapr crate documentation (crates/hairspring-dapr/src/lib.rs). Next: Examples-Overview.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference