Skip to content

[scanner] read-only mode: HealthScanner.upsert() called unconditionally — Liquidatable bucket grows without bound, memory leak and misleading metrics #241

@obchain

Description

@obchain

PR: #51 (feat/23-testnet-config)
File: crates/charon-cli/src/main.rs (drain loop, run_listen), crates/charon-scanner (HealthScanner DashMap)
Refs #51

Problem

When flashloan and liquidator sections are both absent (read-only mode), the drain loop in run_listen still calls:

scanner.upsert(positions);

unconditionally on every block. Positions classified as Liquidatable accumulate in the HealthScanner internal DashMap with no eviction. No liquidation is dispatched, so they are never removed. The scanner has no TTL-based eviction for entries that remain in the Liquidatable bucket across many blocks (flagged in PR #34 review; still unresolved).

Over a multi-hour testnet Grafana demo run:

  • The liquidatable bucket count monotonically increases as distinct borrowers enter the liquidatable range and are never cleared.
  • Prometheus counters report an ever-growing liquidatable count the operator cannot act on.
  • Memory grows proportionally to the number of distinct liquidatable borrowers seen over the session.

Impact

Operational confusion on Grafana dashboard. Memory leak proportional to session length and borrower set size.

Fix

In read-only mode, either:

  1. Skip upsert for positions below the liquidatable threshold: log at debug! and count in a charon_skipped_read_only_total counter.
  2. Add a TTL-based removal policy to HealthScanner (previously flagged, still needed regardless of read-only mode).
  3. At minimum, add a startup gauge charon_run_mode{mode="read_only"} so the dashboard reflects bot state and operators can contextualize the metrics.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglayer:rustRust crates (core / scanner / protocols / executor / cli)priority:p0-blockerBlocks the critical pathstatus:readyScoped and ready to pick up

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions