Skip to content

fix(keeper): drop stale watches and idle duplicate rejections#242

Merged
mfw78 merged 9 commits into
dev/m1from
fix/m1-chassis-sweep
Jul 14, 2026
Merged

fix(keeper): drop stale watches and idle duplicate rejections#242
mfw78 merged 9 commits into
dev/m1from
fix/m1-chassis-sweep

Conversation

@mfw78

@mfw78 mfw78 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What

Fixes two watch-lifecycle bugs in the keeper's cow::run poll loop and adds permanent-drop observability:

  • submit_ready (crates/shepherd-sdk/src/cow/run.rs): an OrderCreation constructor rejection (validTo beyond the client-side one-year horizon, zero from) no longer skips-and-keeps the watch in an unbounded per-block eth_call + Warn loop. It now routes through the Retrier as Drop, matching watch-tower parity. The unknown-enum-marker case keeps its existing skip-and-keep behaviour, since an SDK upgrade could still pick that watch up. New test ready_beyond_the_valid_to_horizon_drops_the_watch asserts zero submits, the watch dropped, and a warn logged.
  • Stop-loss (modules/examples/stop-loss/src/strategy.rs): a DuplicatedOrder rejection is now special-cased ahead of classify_api_error via is_already_submitted. It writes the submitted:{uid} marker the dedup guard reads and idles, instead of re-posting the same order every block until validTo. New regression test duplicate_rejection_records_receipt_and_idles.
  • twap-monitor's poll_one and the keeper run loop's DontTryAgain arm now log the four-byte revert selector and the node's message before a watch is permanently dropped, so a wrongly-dropped watch is triageable from the log alone.
  • Adds a free nexum_sdk::keeper::watch_key helper (no host turbofish needed) and switches call sites (including twap-monitor's test-only watch_key) over to it; drops the now-unneeded shepherd-sdk dev-dependency from shepherd-backtest and nexum-sdk-test.

Why

Two skip-and-keep paths left a watch alive indefinitely once it reached a state that could never resolve: an OrderCreation constructor rejection is deterministic for the polled payload, so keeping the watch just re-polled and re-warned every block forever, and a duplicate-order rejection meant the orderbook already had the order, so re-posting on every subsequent block was pure waste. Both now converge on the watch-tower's actual behaviour (drop on an unrecoverable rejection, idle once the receipt is recorded), and the added selector/message logging means a dropped watch does not need to be reproduced to work out why it was dropped.

Part of the M1 keeper sweep (#135).

Testing

Full gate battery run inside nix develop, mirroring just ci:

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • RUSTDOCFLAGS=-D warnings cargo doc --workspace --no-deps --locked
  • cargo build --release --target wasm32-wasip2 --locked across all example/module targets
  • cargo test --workspace --all-features --no-fail-fast --locked, including the new ready_beyond_the_valid_to_horizon_drops_the_watch, duplicate_rejection_records_receipt_and_idles, and the twap-monitor revert-selector log assertions

AI Assistance

Claude (Opus) used for the implementation.

@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 8d954b8 to e4586a7 Compare July 7, 2026 23:05
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 84ef2ba to 25226ee Compare July 7, 2026 23:05
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from e4586a7 to 25f3387 Compare July 8, 2026 00:42
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 25226ee to ee87527 Compare July 8, 2026 00:42
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 25f3387 to 09486b7 Compare July 8, 2026 01:19
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from ee87527 to 5b05f8e Compare July 8, 2026 01:19
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 09486b7 to f027b50 Compare July 8, 2026 01:44
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 5b05f8e to ef3f9d3 Compare July 8, 2026 01:44
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from f027b50 to 4e2996a Compare July 8, 2026 02:10
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from ef3f9d3 to 850f30c Compare July 8, 2026 02:10
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 4e2996a to 343fe1d Compare July 8, 2026 02:19
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 850f30c to cf6f224 Compare July 8, 2026 02:19
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 343fe1d to 6eabbbc Compare July 8, 2026 02:22
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch 2 times, most recently from cf34ce4 to 8dce752 Compare July 8, 2026 03:03
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch 2 times, most recently from 826abef to 104c6ec Compare July 8, 2026 04:59
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 8dce752 to 3180972 Compare July 8, 2026 04:59
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 104c6ec to 4872819 Compare July 8, 2026 05:30
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 3180972 to 5cbce6a Compare July 8, 2026 05:30
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 4872819 to 8d46460 Compare July 8, 2026 06:21
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 5cbce6a to e8acf48 Compare July 8, 2026 06:21
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 8d46460 to dcc5ee7 Compare July 8, 2026 06:40
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from e8acf48 to a9b18f5 Compare July 8, 2026 06:40
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from dcc5ee7 to c24ac10 Compare July 8, 2026 07:06
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from a9b18f5 to 2763e33 Compare July 8, 2026 07:06
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from c24ac10 to 3b0f42c Compare July 8, 2026 07:20
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 2763e33 to 6cfedbe Compare July 8, 2026 07:21
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 3b0f42c to d90733d Compare July 8, 2026 07:36
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from c36397a to 5179a35 Compare July 8, 2026 12:20
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 2623194 to 2ae623e Compare July 8, 2026 12:49
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 5179a35 to 25ad223 Compare July 8, 2026 12:49
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 2ae623e to 3766202 Compare July 8, 2026 13:40
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 25ad223 to c7af694 Compare July 8, 2026 13:40
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 3766202 to 288c8b4 Compare July 8, 2026 13:59
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from c7af694 to 6b646d4 Compare July 8, 2026 13:59
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 288c8b4 to 38bea45 Compare July 8, 2026 14:08
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from 6b646d4 to 1de11c3 Compare July 8, 2026 14:08

@lgahdl lgahdl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three fixes are correct and the tests are solid. A few items below.

// documented branches from anomalies. NOTE: the "not mirrored"
// skip path was retired with hash-only submission; this
// classification is kept for historical report comparability
// until the harness is reworked around the observer-only

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard still fires on real fixtures: any fixture whose app_data_resolved is None (collected before hash-only submission, or whose hash 404'd) returns RejectedExpected("app_data hash not mirrored") regardless of what the strategy actually did. A fixture that the strategy fails to submit for a completely unrelated reason (watch drop, gate, etc.) will be silently misclassified as expected-skip rather than landing in RejectedUnexpected for triage.

Either remove the guard (the RejectedUnexpected bucket is the right place for it now) or invert the condition to fx.app_data_resolved.is_some() so it only fires for fixtures that actually had resolved data — but the label still needs updating in that case.

PollOutcome::DontTryAgain => watches.remove(watch)?,
PollOutcome::DontTryAgain => {
// The removal is permanent; leave a trace of it even
// for sources that do not log their own outcomes.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Permanent watch removal warrants warn, not info. The comment above correctly says sources may not log their own outcomes — for such a source the only signal that a watch vanished forever is this line, and at info level that's easy to miss in production logs.

Suggested change
// for sources that do not log their own outcomes.
tracing::warn!("{} dropped watch {}", source.label(), watch.key());

}
Err(err) => {
// Success wearing an error status: the orderbook already
// holds this exact order. Record the receipt under the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This correctly writes to dedup_key (submitted:{uid_hex}), but the normal-submit success path a few lines above writes submitted:{server_uid} instead. If the server returns a UID that differs from the client-side uid_hex (the drift path that already logs a warn), the dedup guard reads dedup_key and will not fire next block — the module re-POSTs until validTo. The DuplicatedOrder fix here closes that window correctly, but the root asymmetry in the success path is now visible by contrast. The chassis submit_ready uses the client UID consistently on both paths.

/// stops the submit path downstream ([`gpv2_to_order_data`] fails the
/// same way there); an unsupported chain id does not, so a caller
/// keying idempotency on this value alone re-submits until `validTo`
/// on such a chain - bounded, but callers adding new chains should

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advisory about chain-adders is buried mid-sentence in the None return-value description, making it hard to scan. Consider moving it to a # Note section so the None condition can be read on its own.

Suggested change
/// on such a chain - bounded, but callers adding new chains should
/// on such a chain bounded, but bounded only by `validTo`. New chains
/// should be registered in `cowprotocol::Chain` before go-live.

@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 38bea45 to eca676a Compare July 8, 2026 22:33
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch 2 times, most recently from d5ff2c3 to ce55637 Compare July 8, 2026 23:05
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch 2 times, most recently from 76c778d to 498c843 Compare July 8, 2026 23:24
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch 2 times, most recently from 3b81e00 to b58db49 Compare July 8, 2026 23:29
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 498c843 to 5e1e96a Compare July 8, 2026 23:29
@mfw78
mfw78 force-pushed the fix/m1-chassis-sweep branch from b58db49 to 6ba85d4 Compare July 9, 2026 02:09
@mfw78
mfw78 force-pushed the feat/m1-mock-venue branch from 5e1e96a to 265f467 Compare July 9, 2026 02:09
mfw78 added 9 commits July 14, 2026 23:38
…he order

A constructor rejection (zero from, validTo beyond the client-side
one-year horizon) is deterministic for the polled payload: skipping
with the watch intact re-polled and re-warned on every block forever.
Route it through the retry ledger as a drop, matching the pre-keeper
net effect where the orderbook rejected the shipped body (e.g.
ExcessiveValidTo) and the classifier dropped the watch. The
unknown-enum-marker skip keeps its pinned keep-the-watch behaviour.

Also log the keeper-level DontTryAgain removal so a permanent drop is
observable even for sources that do not log their own outcomes.
classify_api_error maps DuplicatedOrder to TryNextBlock on the
premise that the caller records the submitted: receipt - true for the
run, but stop-loss consumed the classification without the
compensating write, so a duplicate rejection re-POSTed every block
until validTo. Mirror run::submit_ready: treat
already-submitted as success wearing an error status, write the marker
the dedup guard reads, and idle.
…yer dev cycle

WatchSet::key never used the host parameter, forcing a meaningless
turbofish at every call site (one test existed solely to prove two
instantiations agree). keeper::watch_key is the free canon;
WatchSet::key stays as a thin delegate for discoverability.

The keeper acceptance tests only touch the local-store seam, so they
now run against nexum-sdk-test's MockHost (the same MockLocalStore
type), shrinking nexum-sdk's dev cycle from the cross-layer
shepherd-sdk-test pair - which dragged the whole CoW domain layer into
the world-neutral crate's dev graph - to the within-layer pair its own
mock crate documents.
… path

A revert that classifies to DontTryAgain destroyed the watch with only
an Info outcome label; the selector and node message needed to triage
a wrongly-dropped watch were discarded. Warn with both before
returning the outcome, and pin the log lines in the drop test. The
test-only watch_key wrapper now reuses the keeper free fn.
OrderCreation::from_signed_order_data was renamed to OrderCreation::new
in cowprotocol 0.2; and only the unknown-marker case of order_uid_hex
returning None also stops the submit path downstream - an unsupported
chain id does not, so say so instead of claiming both do.
The epic removed resolve_app_data and the orderbook-mock's
GET /api/v1/app_data route, but the replay harness still programmed
that route against a strategy that never requests it. Drop the dead
programming (and the now-unused shepherd-sdk dependency) and refresh
the module doc to the hash-only submission flow. The RejectedExpected
classification is kept for historical report comparability until the
harness is reworked around the observer-only strategy (follow-up).
…atch-channel move

The overview's SDK table predated the epic's headline surface: add the
nexum-sdk keeper row and the shepherd-sdk cow::run row. Amend
ADR-0004 with the move of the [patch.crates-io] channel from
bleu/cow-rs to nullislabs/cow-rs and what the pinned rev carries.
Numeric issue/PR references in .rs files go stale across repository
moves; describe the change instead of pointing at the review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-train Part of the current stacked PR train; lands in sequence, do not merge out of order.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants