Skip to content

v2.0.20 — R1.31: the release gate, with four of this executor's own errors on the record

Choose a tag to compare

@BobbyAxerol BobbyAxerol released this 19 Sep 09:46
· 439 commits to main since this release

v2.0.20 — R1.31: the release gate, with four of this executor's own errors on the record

CERTIFICATE. Digests read with docker image inspect from the running
stack at 2026-09-19 09:20Z; numbers from ledger entry 45 and the R1.31
entries in DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md.

RUNNING IMAGES
projector_v2, projector_v2_2, projector_v2_3,
query_v2_1, query_v2_2, stream_v2_active, stream_v2_passive,
binance_bar_edge
qdl-v2-python:2.0.20-95d9595
sha256:9039236e7a8e570f2364b470b33386ab702bc1dde5ae9d5e7d90a4dda531e8f0
rust_core, rust_core_2, rust_core_3, ingestor_okx_swap
qdl-v2-rust:2.0.19-003b5f9
sha256:b7b9d153f0ed31892007639ca35f42642f9d41e713e4a06a9222fabdd82b81ec
ingestor_binance_usdm
qdl-v2-rust:2.0.18-3ecf0ac
sha256:eec6388421845ef570cb3878145d0c5e0398fc44cc2a662805298295cfb0976a

No image was built for this tag, and none needed to be: git diff from
each image's own commit to HEAD is empty for rust/, Cargo.*, generated/rust,
qdl/ and qdl_sdk/. Everything in this tag is documentation, tests and one
read-only script, so the running images are HEAD's runtime.

Catalog revision 9, source policy 1, authority 1 — none moved.

GATE
Python 1,675 tests, 0 failures, 7 skipped, run against /src in
qdl-v2-python:2.0.20-95d9595. Rust unchanged since 003b5f9, so R1.29's
fmt/clippy/186-test gate still stands.

HISTORY FOR PAST DATES, 14/14 Binance intervals, every one FULL and
compared field by field against the venue's own REST at both ends
1m 431ms/0.1d 5m 444ms/0.4d 15m 323ms/1.3d 1h 395ms/5.0d
4h 265ms/20.0d 6h 269ms/30.1d 12h 107ms/60.4d 1d 98ms/120.4d
3d 100ms/362.4d 1w 58ms/845.4d

DELIVERY TO A CONSUMER, 200 subscribes across five feeds, 0 refused
feed p50 p95 max budget
TRADE 665.5 1024.1 1187.4 3000
QUOTE 587.5 933.7 950.4 2000
MARK_INDEX_PRICE 719.1 1330.1 1373.9 2000
BOOK_SNAPSHOT 655.4 1096.3 1523.3 60000
BOOK_DELTA 847.7 1257.7 1407.5 2000

MARK_INDEX_PRICE holds the thinnest margin in the system, 626 ms against
a BLOCK policy. That is a 2,000 ms budget against a ~650 ms path, not a
defect, and it is named so it is not rediscovered as an incident.

ENDPOINT INVENTORY
216 catalog bindings: 206 live, 0 over their own stale_after_ms, 10 with
no event stored. The ten are the four DNSE bindings — owner decision,
served by V1, and live V2 refuses them with "required data is not
available" — and six spot bindings that no ingestor produces and no
consumer manifest requests. The 53 ingestor subscriptions are 24
binance-usdm and 29 okx, none spot.

CPU BUDGET 5 -> 6 VCORE, on the owner's decision
kafka2 and kafka3 1.25 -> 1.75, rust_core_2 0.50 -> 0.75, stable_redis
unchanged at 0.50. Net ceiling added +1.25. Applied with
docker update --cpus and no container recreated — which is what keeps
stable_redis away from ProjectionCacheMismatch. kafka1 and every role
throttling at roughly zero were left alone.

The raises were first justified on cumulative counters, which is the wrong
basis, and then re-measured with 60-second cpu.stat deltas holding kafka1
and kafka2 fixed as controls. rust_core_2 throttles 0.3% at 0.75 against
3.6% at 0.50 while drawing 0.183 either way — burst shape, so it is kept.
stable_redis throttles 0.0% at both and draws 0.034, 7% of a 0.50 ceiling,
so that raise was withdrawn. The controls moved 0.5-1.0 points on their own,
which sets the noise floor these numbers are read against.

Not proven beneficial and not proven harmful. Summed p95 across the five
feeds fell 6,234 -> 5,642 ms and summed max 7,294 -> 6,443 ms, while
summed p50 rose 3,292 -> 3,475 ms, all under higher ambient load (11.6
against 10.2) from live_data_executor outside this stack. Kept because
the tails improved, the binding margin widened and no consumer was
refused — not because throttling moved. R1.29's C3 cut kafka2 throttling
16.9% -> 4.2% and was still reverted; unlike C3, this raise took nothing
away from another role.

Memory was checked separately, because a cpus limit throttles and cannot
OOM: worst headroom is kafka2 at 50.2% of 2 GiB.

FOUR CORRECTIONS OF THIS EXECUTOR'S OWN REPORTS

  1. A regression I caused. The B1 roll recreated ingestor_okx_swap with a
    chain carrying r125-rollout.override.yml but not
    okx-ingestor-image.override.yml, dropping it from 2.0.19-003b5f9 to
    2.0.17-1acf87a — the digest that override names as its own rollback.
    Repaired; the rendered diff was exactly one line.
  2. "Sixteen bindings are over budget" — no. I compared the spool's
    committed_at_ns against stale_after_ms; the runtime reads a
    PROVIDER_CONFIRMATION binding from received_at_ns. Correct: 0.
  3. "TRADE stream p50 1530 ms, a regression" — it was the drain after a
    projector cold start. Steady state 619 ms.
  4. "The raise cut kafka2 throttling 14.4% -> 1.4%" — an invalid
    comparison of a 47-hour cumulative counter against a 120 s window.
    kafka1, untouched, read 1.2% in the same window. Withdrawn, and the
    four raises were re-measured with 60-second deltas; one of them,
    stable_redis, turned out to fix nothing and was reverted.

CLOSING STATE, 09:20Z
17/17 roles Up, 14 healthy; the three rust_core replicas carry no
healthcheck, which needs a Rust change and is recorded with B3.
206 bindings live, 0 over budget, 0 refused in 200 subscribes.
Data layer draw 3.76-4.29 vcore against the new 6.0 budget; per-role
60-second draw kafka2 0.796, kafka3 0.666, kafka1 0.481, rust_core_2 0.183,
stable_redis 0.034.
Images 33 -> 29, build cache 6.60 -> 5.61 GB, no volume and no container
removed — the three dangling volumes are the three kept deliberately and
the only two stopped containers are the stack's own one-shot inits.

RUNTIME HYGIENE CLOSED
This stack had no single recorded compose chain: each container stores the
chain used the last time it was created, and they disagreed — okx ingestor
19 files, the B1 roles 18, rust_core_2 15, kafka2 10, stable_redis 1 with a
base compose file in a worktree that no longer exists. Recreating a role
from its own label drops every override added since, which is exactly how
the OKX ingestor fell back an image today. canonical-chain.txt now records
the complete chain; rendering it and comparing image, cpus and healthcheck
against docker inspect for all seventeen roles gives 0 mismatch.

Two prohibitions are written with it: never up -d without --no-deps,
since every stored config hash came from a shorter chain; and never recreate
stable_redis, whose base file is gone and whose recreation freezes the spool.

NOT IN THIS RELEASE
Debt B as a block: B2 bar-edge active/passive, B3 Rust ingestor HA
(lease_epoch is a static fencing token, not leader election), B4 Redis HA.
One measured optimisation deferred with them: each projector posts every
canonical batch to both stream gateways and the non-holder answers 409,
about 36,000 rejected mTLS POSTs an hour across three.