No Blocking DB Hot Path

No Blocking DB on the Hot Path

The order hot path in Emporia holds one rule: nothing on the single writer thread may block on PostgreSQL — not a read, not a write, not a lock.

This page explains why that rule exists, the five mechanisms that implement it, and — just as importantly — the three constraints Emporia accepts in exchange. The trade is real and the costs are listed here rather than buried.

---

📌 Why: the database was the writer's work

OrderStateCache performs two lookups on every order:

lookup	purpose
existsById	duplicate-order guard
findProcessedById	idempotency (Idempotency-Key)

Both used to read through to PostgreSQL. They were not "mostly misses" — they were zero cache hits across 14,402 orders, because a CREATE carries freshly generated identifiers, so the read-through fired every single time.

Together they cost 2.312 ms of the handler's 2.405 ms.

Why that is structural, not merely slow

The cost sits on the single writer thread. A fixed per-event cost there does not add to latency — it multiplies, because every command pays it and every command queued behind it waits.

The same effect was measured independently with tracing: 0.6 ms per command at 120 orders/sec surfaced as several hundred milliseconds of ring queue wait.

Six alternating runs at 120 orders/sec, with the in-memory index armed and disarmed:

	ring queue wait (mean)	max	k6 submit p99
index on	0.035 – 0.230 ms	17 – 52 ms	27 – 73 ms
index off	1.872 – 5.712 ms	248 – 479 ms	106 – 361 ms

---

🏗️ How: five mechanisms

flowchart TD
    REST["POST /api/orders"] --> RING["LMAX Disruptor ring<br/>single writer thread"]

    subgraph HOT ["Hot path — no PostgreSQL call"]
        RING --> IDX{"RotatingDedupIndex<br/>'certainly never seen'?"}
        IDX -->|"never seen<br/>(from memory)"| APPLY["Apply state transition"]
        IDX -->|"possibly seen"| CACHE{"Caffeine tiers"}
        CACHE -->|hit| APPLY
        APPLY --> WAL["MemoryMappedWalLogger<br/>append, no fsync — 0.3%"]
        WAL --> ACK["201 returned"]
    end

    CACHE -->|miss| PG[("PostgreSQL<br/>exact answer")]
    PG --> APPLY
    ACK --> Q["AsyncDbWriter queue"]
    Q -->|"batch flush, ~10 ms"| PG

    style HOT fill:#0b3d2e,stroke:#12805c,color:#e8f5ef
    style PG fill:#2b2b40,stroke:#5b5b8a,color:#e8e8f5

Loading

1. One writer thread, an LMAX Disruptor ring. All order commands are applied in sequence by a single thread — no locks, no database-level serialisation. This is what makes the rule enforceable: there is exactly one place to keep clean.

2. RotatingDedupIndex answers "certainly never seen" from memory. A set of Bloom filters, rotated at each trading-session start. It replaces both hot-path reads. A Bloom filter cannot produce a false negative, so "never seen" is trustworthy; "possibly seen" falls through to the database, which gives the exact answer.

3. Two Caffeine tiers — not interchangeable with the index. trading-orders returns the order object that modify, cancel and fill mutate; a Bloom filter cannot, it answers a boolean. processed-commands returns the recorded result so a retry gets its original 201.

4. A memory-mapped write-ahead log covers the acknowledged-but-unwritten window. append writes into a mapped region and does not force. It costs 0.011 ms of the handler's 3.047 ms — 0.3%.

5. AsyncDbWriter batches writes off the hot path. The order is answered 201 once the ring has applied it; the row reaches PostgreSQL on a flush, by default every 10 ms.

⚠️ This inverts the usual LMAX advice. The standard recommendation is to move journalling to a parallel handler. Here that would buy 0.3%. The database reads were the cost, not the journal.

---

🎯 What actually moved into memory

Precisely one answer: "this identifier has certainly never been seen."

Everything else — every positive answer, every order object, every recorded result — still comes from PostgreSQL. PostgreSQL is the record; memory is what the system acts on, and for that single answer, what it trusts.

The narrowness is the point. It is the only answer where being wrong in the safe direction costs one database lookup, and being wrong in the unsafe direction is impossible from the filter itself.

---

⚖️ The price: three constraints

These are the terms of the trade, not caveats. Each is enforced or instrumented rather than assumed.

#	constraint	how it is handled
1	Exactly one instance may accept orders	Two instances would each believe the same identifiers are new. DisruptorOrderPipeline refuses with 503 when isPrimary() is false. Limit: no leader-election provider in the repository excludes a second machine.
2	A lost durable write is silent	The hot path no longer reads PostgreSQL, so nothing on it would notice an empty database. This is why the counters below exist.
3	The WAL recovers process death, not machine loss	Mapped pages belong to the OS, so they survive kill -9 and a JVM crash — not the machine dying. At 120 orders/sec with a 10 ms flush, the exposure is one or two orders already answered 201, possibly already at the venue, and nothing notices.

The remedy for (3) is known and deliberately not taken: append on the writer thread, have a separate thread force, and only then complete the HTTP response — group commit with a durable acknowledgement. It never loses an acknowledged order, and it adds the force interval to every submit. That is precisely the blocking call this rule exists to avoid. Revisit if the requirement becomes "an acknowledged order survives machine loss".

---

🔍 The oracles

Because the hot path cannot notice a database problem, three counters let the database speak from the far side of it. All three must stay at zero.

counter	fires when
emporia.oms.dedup.duplicate_reached_db	a duplicate command reached the writer
emporia.oms.dedup.duplicate_order_reached_db	a duplicate order id reached the writer
emporia.oms.writer.rejected_rows	the database refused a row

They are database-side by design. An in-memory check that the in-memory index is working would be arguing with itself.

---

🧪 Evidence, and what it does not cover

Two hours at 10 orders/sec, one request in ten replaying an earlier Idempotency-Key:

orders submitted	72,001
duplicate commands sent	~7,160
all three counters	0
lookups answered from the index	131,836
lookups that reached PostgreSQL	10 (0.0076%)

Those ten are the designed fall-through, not a fault: a replay whose recorded result had been evicted from the Caffeine tier reached the filters, which answered "possibly seen", and the database returned the original result.

Reproducible checks:

script	asserts
scripts/perf/crash-recovery-check.sh	unflushed orders are replayed after kill -9
scripts/perf/wal-recovery-check.sh	a command in flight at the kill is still deduplicated after restart
scripts/perf/dedup-horizon-check.sh	a repeated key deduplicates inside the horizon and not past it

Warm-up costs latency, not correctness. Orders are accepted from the first moment and answered by PostgreSQL until the startup load finishes — over a 20,001-row table, 292 ms.

❗ What this is not evidence for. The soak ran on a development machine, once, at a tenth of the benchmarked rate, with a synthetic retry pattern that only replays recent keys. It says nothing about latency — and nothing about machine loss, which no test here exercises.

---

🗺️ Where to look in the code

concern	class
single writer, primary check	DisruptorOrderPipeline
memory-authoritative "never seen"	RotatingDedupIndex, CommandDedupIndex
when filters rotate	RotationSchedule, DedupIndexRotation
three-tier lookup	OrderStateCache
batched writes, conflict absorption	AsyncDbWriter
crash window	MemoryMappedWalLogger
startup load	DedupIndexWarmup, DedupIndexLoader

---

See also: Architecture & Order Flow · WAL Crash Recovery · Order Management Service