Skip to content

Example Dapr orders

Tom Barber edited this page Oct 25, 2026 · 1 revision

Example: Dapr orders

A small Python service that must apply each order exactly once, tested against a simulated Dapr sidecar that injects duplicate deliveries, reordering and app crashes from a seed. It is the easiest of the process-style examples to try, because the simulator can run your app directly, with no Hermit, on a Mac or on Linux.

Files: examples/dapr-orders. Background: the Dapr section of the README and docs/determinism.md.

Goal

See a crash-recovery bug appear for some seeds and not others, see the fixed consumer survive the same faults, and understand how the same command is used as a process target under Hermit.

The bug

buggy_app.py deduplicates redelivered messages with an in-memory set. That works until the app restarts: the set is lost, the broker delivers the order again (Dapr pub/sub is at-least-once), and the order is applied twice. The Dapr state store survives the restart; the app's memory does not.

fixed_app.py deduplicates with a first-write-wins marker in the state store, so it survives restarts. Everything else is identical.

Prerequisites

Need For
A Rust toolchain building the simulator, hairspring-dapr-sim. It is not part of the release binaries
Python 3 the app (standard library only)
A clone of the repository the example files
For the Hermit part only: Linux x86-64 and Hermit (Installation) exploring seeds under Hermit

Steps

1. Build the simulator

cargo build --release -p hairspring-dapr

This puts target/release/hairspring-dapr-sim in the clone. It took a few seconds on our Mac once the workspace had been built. Its only documented option set is:

hairspring-dapr-sim: usage: hairspring-dapr-sim [--scenario FILE] [--seed N] [-- APP COMMAND...]

2. Look at the scenario

scenario.toml is the workload and fault plan:

[app]
env = { EXPECTED_IDS = "1,2,3", PYTHONDONTWRITEBYTECODE = "1", PYTHONPYCACHEPREFIX = "/hairspring-empty-pycache", PYTHONHASHSEED = "0" }

[faults]
duplicate = 0.5
reorder = 0.4
crash_app = 0.3

[[publish]]
topic = "orders"
data = { id = 1 }
# ... and orders 2 and 3

Three orders are published. For each delivery, the simulator draws from a seeded random number generator: with probability 0.5 it redelivers a message that was already processed, with 0.4 it delivers out of order, and with 0.3 it kills and restarts the app after a delivery, losing its memory. When the run ends the simulator asks the app to check itself (GET /hairspring/check); the app answers 500 with order N was applied M times if any order was not applied exactly once. The simulator turns that into a line starting HAIRSPRING_VIOLATION:.

The other fault types the simulator supports are drop, state_conflict and state_error. Only an implemented subset of Dapr is simulated: state with ETags and first-write concurrency, publish, and subscription delivery. No bindings, secrets, actors, workflows or gRPC.

3. Run it directly, with a seed

From examples/dapr-orders:

../../target/release/hairspring-dapr-sim --scenario scenario.toml --seed 1 -- python3 buggy_app.py
echo $?

Expected output and exit status (we ran this):

HAIRSPRING_VIOLATION: order 1 was applied 2 times

and exit status 1. Now a seed that passes:

../../target/release/hairspring-dapr-sim --scenario scenario.toml --seed 2 -- python3 buggy_app.py
hairspring-dapr-sim: ok (seed 2): 4 delivered, 1 redelivered, 0 crashes

with exit status 0. The repository's README suggests seed 3 for the first command. On the Mac we used, the buggy app passes at seed 3, so use seed 1 to see the failure. (The README says "run it directly (no Hermit, so faults are seeded but thread timing is not)", which is the point: without Hermit the faults repeat for a seed but the app's own thread timing does not.)

4. Try several seeds

We ran seeds 1 to 12 against the buggy app, and 1 to 8 against the fixed one:

Seed buggy_app.py fixed_app.py
1 violation: order 1 applied 2 times ok (2 crashes)
2 ok ok
3 ok ok
4 violation: order 1 applied 2 times ok (3 crashes)
5 violation: order 2 applied 2 times ok (2 crashes)
6 violation: order 3 applied 2 times ok (2 crashes)
7 ok ok (1 crash)
8 ok ok
9, 10 violation not run
11, 12 ok not run

Run the same seed twice and you get the same answer (we repeated seeds 1 and 4 and saw the same output). That is the sidecar's fault plan being seeded. It does not show that the app's thread timing is repeatable without Hermit.

To run the fixed app: replace buggy_app.py with fixed_app.py in the command.

5. Read a result

For a buggy_app.py seed that failed, the line names the broken invariant: which order, applied how many times. In the passing line, 4 delivered, 1 redelivered, 2 crashes is the fault plan's work for that seed: the number of delivery attempts, how many were redeliveries, and how many times the app was killed. A passing seed with 0 crashes says little, because the bug needs a crash between a delivery and its redelivery. A seed that fails is more informative than a seed that passes.

Compare the two apps on a seed where the fixed app saw crashes (seeds 1, 4 and 5): it survived several restarts and the order was still applied once.

6. As a process target under Hermit (Linux x86-64)

Everything in this step comes from the example's README. We could not run it on a Mac. In hairspring.toml:

[target.orders-buggy]
kind = "process"
cwd = "/path/to/examples/dapr-orders"
command = ["/path/to/hairspring-dapr-sim", "--scenario", "scenario.toml", "--", "/usr/bin/python3", "buggy_app.py"]

[target.orders-buggy.budget]
max_trials = 40

then hairspring test orders-buggy. The Hermit backend sets HAIRSPRING_SEED to each trial's seed, and the simulator uses it to draw its faults. So the seed now controls both the faults and Hermit's thread scheduling. Add hairspring test orders-buggy --sweep --trials 40 for a rate, and make a second target for fixed_app.py to compare.

The project's determinism notes record, from earlier testing on a Linux machine, that the buggy consumer failed with order 1 was applied 2 times at sched-seed 1 and repeated at that seed 3 of 3 times, and that the fixed consumer passed 12 of 12 seeds. We did not repeat that run. Treat it as a record of one setup, not a rate.

What happens on a Mac if you try step 6 anyway (we ran this):

hairspring: exploring orders-buggy (seed 1)
ERROR   orders-buggy: hermit executable not found at `hermit`. Hermit runs only on x86-64 Linux: install it there, or set HAIRSPRING_HERMIT to its path

The command exits with status 2. See Troubleshooting.

What to look at

  • buggy_app.py: the seen = set() # BUG: lost on restart line, and first_time().
  • fixed_app.py: first_time() writing a seen-<id> marker with "options": {"concurrency": "first-write"}, and returning true (first time seen) only when the state store answers 204.
  • scenario.toml: the PYTHON* variables freeze Python's bytecode caches. They are there because Python rewrites .pyc files, which changes what runs next time; see "Hidden inputs" in docs/determinism.md.

Cleanup

Nothing is installed. The simulator writes nothing into the example directory (we checked git status). Remove the build output with cargo clean -p hairspring-dapr if you wish.

Where the full write-up lives

The Dapr section of the README, examples/dapr-orders, and the hairspring-dapr crate documentation (crates/hairspring-dapr/src/lib.rs). Next: Examples-Overview.

Clone this wiki locally