feat(backtest): pre-soak EthFlow replay harness (COW-1078)#60
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
## First run
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
## Scope
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
## Tests
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
## Reproducing
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end
and validated against a live Sepolia 7d window before push.
2 tasks
Author
This was referenced Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three-piece harness that takes the last N days of Sepolia events and replays them through the production strategy code offline — derisks the COW-1031 soak by moving strategy-bug discovery from days of wall-clock to minutes of CPU.
tools/backtest-collect/backtest_collect.py— Python collector. Pulls trailing-N-dayOrderPlacement(EthFlow) andConditionalOrderCreated(TWAP) events from Sepolia, ABI-decodes payloads, derives EthFlow OrderUids via EIP-712 against the chain's GPv2Settlement domain, resolves every non-emptyappDatahash viaGET /api/v1/app_data/{hash}, dumps a single fixtures JSON. Reuses log-scan + UID-derive code from the baseline-latency tool (COW-1084, PR feat(ops): orderbook EthFlow indexer baseline tool (COW-1084) #57).crates/shepherd-backtest— Rust binary. Loads the fixtures, programs aMockHostper event (resolved app_data + UID-echo submit), drivesethflow_watcher::strategy::on_logsdirectly. Classifies intoSubmitted/RejectedExpected/RejectedUnexpected/StrategyErrorand renders the report.modules/ethflow-watcher—crate-type = ["cdylib", "rlib"]+ cfg-gate the wit-bindgen glue. The rlib carries only thepub mod strategyfor native consumers; the wasm artefact is unchanged.First run
7-day window (2026-06-15..2026-06-22) on Sepolia: 240 EthFlow events → 240 Submitted, 0 anomalies = 100.0% vs. the 95% sign-off threshold. Report committed at
docs/operations/backtest-reports/backtest-7d-2026-06-22.md.26 TWAP
ConditionalOrderCreatedevents collected and counted, but the replay is deferred to Phase 2B — drivingtwap_monitor::strategy::on_blockneeds per-blocketh_call(getTradeableOrderWithSignature)walking, which public-tier RPCs refuse (independently confirmed by the baseline-latency tool). The fixtures are committed for the future paid-RPC re-run.Why this matters for the grant
Without a backtest, the soak risk is: start COW-1031, burn N days of wall-clock, hit a strategy bug on day N, fix it, restart the 7-day clock. The backtest moves that discovery to ~5 seconds of replay over 240 real events. The 100% pass result here clears the soak's backtest gate.
Scope
POST /api/v1/quotevalidation on captured bodiesTests
cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets -- -D warningscleanStacked on
PR #59 (
feat/backoff-retry-cap-cow-1083) → PR #58 → PR #57 → upstream M4 epic (TBD).AI-assistance disclosure
Authored with Claude (Opus 4.7); reviewed end-to-end and validated against a live Sepolia 7d window before push.