Skip to content

feat(backtest): pre-soak EthFlow replay harness (COW-1078)#60

Closed
brunota20 wants to merge 1 commit into
feat/backoff-retry-cap-cow-1083from
feat/backtest-cow-1078
Closed

feat(backtest): pre-soak EthFlow replay harness (COW-1078)#60
brunota20 wants to merge 1 commit into
feat/backoff-retry-cap-cow-1083from
feat/backtest-cow-1078

Conversation

@brunota20

Copy link
Copy Markdown

Summary

Three-piece harness that takes the last N days of Sepolia events and replays them through the production strategy code offline — derisks the COW-1031 soak by moving strategy-bug discovery from days of wall-clock to minutes of CPU.

  1. tools/backtest-collect/backtest_collect.py — Python collector. Pulls trailing-N-day OrderPlacement (EthFlow) and ConditionalOrderCreated (TWAP) events from Sepolia, ABI-decodes payloads, derives EthFlow OrderUids via EIP-712 against the chain's GPv2Settlement domain, resolves every non-empty appData hash via GET /api/v1/app_data/{hash}, dumps a single fixtures JSON. Reuses log-scan + UID-derive code from the baseline-latency tool (COW-1084, PR feat(ops): orderbook EthFlow indexer baseline tool (COW-1084) #57).
  2. crates/shepherd-backtest — Rust binary. Loads the fixtures, programs a MockHost per event (resolved app_data + UID-echo submit), drives ethflow_watcher::strategy::on_logs directly. Classifies into Submitted / RejectedExpected / RejectedUnexpected / StrategyError and renders the report.
  3. modules/ethflow-watchercrate-type = ["cdylib", "rlib"] + cfg-gate the wit-bindgen glue. The rlib carries only the pub mod strategy for native consumers; the wasm artefact is unchanged.

First run

7-day window (2026-06-15..2026-06-22) on Sepolia: 240 EthFlow events → 240 Submitted, 0 anomalies = 100.0% vs. the 95% sign-off threshold. Report committed at docs/operations/backtest-reports/backtest-7d-2026-06-22.md.

26 TWAP ConditionalOrderCreated events collected and counted, but the replay is deferred to Phase 2B — driving twap_monitor::strategy::on_block needs per-block eth_call(getTradeableOrderWithSignature) walking, which public-tier RPCs refuse (independently confirmed by the baseline-latency tool). The fixtures are committed for the future paid-RPC re-run.

Why this matters for the grant

Without a backtest, the soak risk is: start COW-1031, burn N days of wall-clock, hit a strategy bug on day N, fix it, restart the 7-day clock. The backtest moves that discovery to ~5 seconds of replay over 240 real events. The 100% pass result here clears the soak's backtest gate.

Scope

  • v1 (this PR): EthFlow lane end-to-end (collector + replay + report)
  • Phase 2B (follow-up): TWAP lane via paid-RPC archive walking
  • Phase 2C (follow-up): downstream POST /api/v1/quote validation on captured bodies
  • Out of scope per issue: supervisor / WS reconnect (stays on soak); fuel/memory limits (stays on COW-1036); orderbook PUT mutation (forbidden by design)

Tests

  • 19/19 ethflow-watcher tests pass (rlib + cdylib both clean)
  • Full workspace test sweep passes (no regressions to engine, twap-monitor, SDK)
  • cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets -- -D warnings clean
  • Live run on Sepolia: 240 fixtures → 240 Submitted, 0 anomalies

Stacked on

PR #59 (feat/backoff-retry-cap-cow-1083) → PR #58 → PR #57 → upstream M4 epic (TBD).

AI-assistance disclosure

Authored with Claude (Opus 4.7); reviewed end-to-end and validated against a live Sepolia 7d window before push.

Adds the COW-1078 pre-soak backtest end-to-end:

1. `tools/backtest-collect/backtest_collect.py` — Python collector
   that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
   and `ConditionalOrderCreated` (TWAP) events from Sepolia,
   ABI-decodes each payload, derives the EthFlow `OrderUid` via
   EIP-712 against the chain's GPv2Settlement domain, resolves every
   non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
   emits a single fixtures JSON. Reuses the log-scan + UID-derive
   infra introduced by the baseline-latency tool (COW-1084 PR #57).

2. `crates/shepherd-backtest` — new Rust binary that loads the
   fixtures, programs a `MockHost` per event (resolved `app_data`
   response + UID-echo submit response), and drives
   `ethflow_watcher::strategy::on_logs` directly. Each event is
   classified into `Submitted` / `RejectedExpected` /
   `RejectedUnexpected` / `StrategyError` and rendered into a
   Markdown report at `docs/operations/backtest-reports/
   backtest-7d-YYYY-MM-DD.md`.

3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
   and cfg-gate the wit-bindgen glue so the rlib carries only the
   `strategy` module (now `pub mod`) for native consumers. The
   wasm artefact is unchanged.

## First run

7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.

26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.

## Scope

- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
  validation via `POST /api/v1/quote` round-trip on captured
  bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
  coverage (stays on the wall-clock soak); fuel/memory limits (stays
  on COW-1036 / soak); orderbook PUT mutation (forbidden — only
  read-only endpoints are touched).

## Tests

- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
  -- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies

## Reproducing

```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
    --fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```

AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end
and validated against a live Sepolia 7d window before push.
@linear-code

linear-code Bot commented Jun 22, 2026

Copy link
Copy Markdown

COW-1078

@brunota20

Copy link
Copy Markdown
Author

Work landed via dev/m5-base creation @ 04381d3 (M5 packaging + PR #67 rust-idiomatic compliance + PR #68 docs reconciliation squashed). The HEAD commit of this PR is now an ancestor of dev/m5-base. Closing as merged-by-ancestor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant