feat(mempool): observability for the TxMempool rewrite invariants

[bdchatham]

Adds two metrics that expose the TxMempool rewrite's compaction behavior without changing logic: tendermint_mempool_compact_total{trigger} and tendermint_mempool_compact_duration_seconds. The trigger label distinguishes the three call sites of compact() (insert_overflow, update, reap); rate-of-insert_overflow is the capacity-pressure signal. Emitted via OpenTelemetry (otel.Meter("tendermint_mempool")) following the sei-db convention.

Companion to platform sei-protocol/platform#743 (recording rules, dashboard panels, alerts). The reactor-side gossip-bytes counter and any pending→ready promotion signal are separate followups — the latter is already directly tested by mempool_pending_promotion_test.js in the release-test suite, so a metric for it isn't yet pulling weight.

Metric rationale

These metrics let us answer four questions about the rewrite that we couldn't ask cleanly before.

What each metric is for, when stress-testing

compact_total{trigger="insert_overflow"} is the capacity-pressure signal. When this counter ticks, the mempool was momentarily over hardLimit (= 2× softLimit). A non-zero rate means ingress is outpacing block consumption — the only operational question is whether admission control is supposed to be catching it. If both check_tx_met_drop_utilisation_threshold and insert_overflow are rising, pressure escaped the gate.

compact_total{trigger="update"} is the block heartbeat. It fires once per Update, so its rate is a block-rate proxy. If it flatlines while seiload is still pushing, the consensus loop is stuck — different signal from block_height_delta because it's measured at the mempool, not the indexer.

compact_total{trigger="reap"} is this node's proposer share. Validators that propose blocks Reap; followers don't. Useful for telling "is this node doing block-proposal work?" without consulting Tendermint internals.

compact_duration_seconds is where we'll catch the rewrite's real failure mode. compact() is O(m log m) over the full mempool. As load grows, this latency grows. The number we'll watch is rate(compact_total[5m]) × avg(compact_duration_seconds) — wall-time-fraction spent inside compact. Once that approaches ~50%, compact is now blocking block production. The histogram lets us see this coming long before p99 actually hits block-interval (~200ms).

The composite questions only these together can answer

1. Does the rewrite's amortization actually hold under load? rate(insert_overflow) / rate(inserted_txs) should stay sub-linear as throughput rises. If the ratio goes to 1, the 2× hardLimit amortization broke and we're back to per-insert prune.
2. Is compact CPU-bound or block-bound? Compare per-call duration to per-block compact rate. compact_duration_p99 rising independently of insert_overflow rate means individual compacts got expensive (likely GC pressure on 100k+ entries); both rising means we're pruning more often AND each prune is slower — the failure mode.
3. Is admission control doing its job or just delaying the problem? rate(check_tx_met_drop_utilisation_threshold) / rate(insert_overflow) — if it's high, the gate is absorbing the spike. If it's near zero while overflow is high, the gate's threshold is mistuned.

What we're still blind to

These metrics don't tell us why a compact was slow (sort vs. map-rebuild vs. GC) — that needs pprof or per-phase timing inside compact(). Per-peer gossip bandwidth is the next PR. The pending→ready promotion behavior is directly tested by mempool_pending_promotion_test.js; a continuous prod signal for it can be added later if operational need surfaces.

Test plan

• go build ./... clean
• go test ./internal/mempool/ -count=1 passes
• Validate emissions in a nightly chaos run
• Wire up platform-side recording rules + dashboard panels per do not unregister if out of rent #743

References: design doc sei-protocol/platform#743, TxMempool rewrite #3476.

🤖 Generated with Claude Code

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	May 29, 2026, 3:01 PM

[bdchatham]

Coral cross-review applied (commit c92945f)

Both platform-engineer and observability-platform-engineer reviewed the initial commit. All must-fix findings landed in this revision:

Platform-engineer findings (Utilisation gauge staleness):

• Flush() cleared the store but left gauges stale → set Size/PendingSize/TotalTxsSizeBytes/Utilisation to 0 after Clear().
• ReapTxs(remove=true) evicts via compact("reap") but never re-emits the gauges → set them at end of ReapTxs when remove=true. (Same bug class existed for the pre-existing Size gauge — fixed together.)

Observability findings:

• compact_total{reason} → compact_total{trigger} — Prometheus convention bias; reason reads as failure cause, trigger reads as caller dimension. Free pre-scrape; expensive after.
• promotion_total → promotion_total{tx_type="evm"} — locks the EVM scope into a queryable dimension instead of metric-name lore; preserves the door for tx_type="cosmos" later without renaming.
• compact_duration_seconds buckets extended to …, 5, 10, 20, 30 — the docstring set expectation of 5-10s under GC pressure, but the upper bucket of 10s meant p99 would saturate at +Inf exactly when needed most.

Non-blocking findings (deferred):

• One-line panic-defer comment (cosmetic)
• Hoisting .With() outside the defer (micro-opt; compact is per-block, not nanosecond-hot)

Bug found in the consumer-side platform PR #743: several PromQL queries reference Go-case names (tendermint_mempool_PendingSize, tendermint_mempool_Size) but the actual scrape names are snake_case (pending_size, size). Filing a fix-up to platform#743 separately.

Ready for human review when convenient.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.22%. Comparing base (8567422) to head (bd7f503).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3519      +/-   ##
==========================================
- Coverage   59.03%   58.22%   -0.82%     
==========================================
  Files        2199     2129      -70     
  Lines      182207   174096    -8111     
==========================================
- Hits       107569   101363    -6206     
+ Misses      64975    63729    -1246     
+ Partials     9663     9004     -659     

Flag	Coverage Δ
sei-chain-pr	73.54% <100.00%> (?)
sei-db	70.41% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-tendermint/internal/mempool/mempool.go	81.86% <100.00%> (+0.74%)	⬆️
sei-tendermint/internal/mempool/metrics.go	100.00% <ø> (ø)
sei-tendermint/internal/mempool/tx.go	92.85% <100.00%> (+0.22%)	⬆️

... and 70 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

Low Risk
Observability-only changes around existing compact() and gauge updates; no mempool admission, ordering, or consensus logic is modified.

Overview
Adds OpenTelemetry instrumentation around the TxMempool rewrite’s compact() path without changing compaction logic. New signals are tendermint_mempool_compact_total (with trigger = insert_overflow, update, or reap) and tendermint_mempool_compact_duration_seconds, recorded at the three existing call sites in txStore (Insert overflow, Update, Reap with removal).

TxMempool now keeps existing go-kit gauges in sync when the store is cleared or txs are reaped: Flush zeros size gauges, and ReapTxs refreshes ready/pending/byte gauges when remove is true (matching post-CheckTx / Update behavior).

^{Reviewed by Cursor Bugbot for commit bd7f503. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bdchatham]

Both reviewer comments addressed:

@cursor[bot] — fixed the PromotionTotal double-counting in compact() (commit `d5d4a1a`). The metric was incrementing for every re-inserted EVM tx during compact's account.nextNonce = firstNonce reset, so the rate would have been dominated by mempool size × block rate instead of new external promotions. insert() now takes a countPromotion bool flag that the public Insert() sets to true and compact()'s re-insertion path sets to false. The PR-description invariant rate(promotion_total) ≈ rate(inserted_txs) now holds.

@amir-deris — converted the 3 new metrics to OTel native (commit `105bfd2`). Following the sei-db convention (sei-db/db_engine/pebbledb/mvcc/metrics.go):

var (
    mempoolMeter = otel.Meter("tendermint_mempool")
    otelMetrics = struct {
        compactTotal           metric.Int64Counter
        compactDurationSeconds metric.Float64Histogram
        promotionTotal         metric.Int64Counter
    }{ /* ... */ }
    triggerInsertOverflowAttr = metric.WithAttributes(attribute.String("trigger", "insert_overflow"))
    /* ... */
)

// Emit site:
otelMetrics.compactTotal.Add(context.Background(), 1, triggerInsertOverflowAttr)
otelMetrics.compactDurationSeconds.Record(context.Background(), time.Since(start).Seconds())

Attribute sets are package-level vars so emission is allocation-free on the hot path. The existing 22 go-kit metrics in this package remain untouched — they should move to OTel in a follow-up that handles the whole package consistently rather than mixing per-metric.

[bdchatham]

Taking a follow-up to properly pass context in

[bdchatham]

Final Coral review round — one must-fix landed

Both experts re-reviewed at `04eee9659`. Platform-engineer returned "ready"; observability caught a real blocker.

Naming mismatch — fixed in `a36521af`. The OTel→Prom exporter in sei-chain registers with `namespace="sei-chain"` (`utils/metrics/metrics_util.go`), so the meter name ("tendermint_mempool") becomes the `otel_scope_name` label, not a series prefix. With the short names `compact_total` / `compact_duration_seconds`, the exported series would have been `sei_chain_compact_total` — the platform PR #743's recording rules expecting `tendermint_mempool_compact_total` would have silently returned no data.

Fix: prefixed the metric names with `tendermint_mempool_` so the exported series is `sei_chain_tendermint_mempool_compact_total`. Matches the existing precedent (`sei_chain_app_tx_count_total` from `meter="app"`, metric="tx_count"`) and sits alongside the existing go-kit `tendermint_mempool_*` metrics in queries.

Platform appendix in sei-protocol/platform#743 updated in parallel (`f39756b`) to reference `sei_chain_tendermint_mempool_*` in PromQL.

Other findings — all green:

• Histogram bucket spec preserved (14 boundaries + `+Inf` + `_count` + `_sum` = 17 series/instance)
• Cardinality recount: ~5,000 series net-new
• Mixed go-kit + OTel coexists cleanly on the same `/metrics` endpoint
• Attribute-set pre-baking is safe (no aliasing)
• OTel hot-path emit cost ~50-200ns/call, comparable to go-kit
• Bugbot's pre-fix finding was a real bug we should have caught in round 1

Ready for human review.

[bdchatham]

@pompon0 — addressed in `bd7f50392`:

1. `utils.OrPanic1` — swapped the local `must[V any]` helper for the shared one.
2. Exponential buckets — switched to `stdprometheus.ExponentialBucketsRange(0.001, 30, 14)` (the same helper used in `internal/consensus/metrics.gen.go`). 14 boundaries spanning 1ms to 30s with a ~2.32× factor per bucket.
3. Timing inside `compact()` — keeping it at the call sites. `compact()` stays pure (no OTel emission inside) and the per-`trigger` counter increment is already at the caller, so the timing sits naturally next to it. The duplication is 2 lines × 3 sites and visually clear; moving timing inside would either require a label parameter (which the prior review pass cut) or split the histogram/counter emission across two locations.

[amir-deris]

It probably doesn't need _seconds in the name. That will be added by default.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bd7f503. Configure here.}

[cursor]

Duration histogram missing trigger attribute at all call sites

Medium Severity

The compactDurationSeconds.Record() calls omit the trigger attribute (triggerInsertOverflowAttr, triggerUpdateAttr, triggerReapAttr) that is passed to compactTotal.Add(). All three call sites record duration into an unlabeled histogram, making it impossible to break down compact latency by trigger. This defeats the stated justification for keeping timing outside compact() ("The dimension of the metric depends on where this is called").

Additional Locations (2)

• sei-tendermint/internal/mempool/tx.go#L533-L534
• sei-tendermint/internal/mempool/tx.go#L600-L601

 

^{Reviewed by Cursor Bugbot for commit bd7f503. Configure here.}