feat(gossip): add semantic dedupe and ledger mode

[Lythaeon]

Description

This PR is the stacked follow-up on top of feature/jito-transport.

It does not add new Jito transport logic. It covers only the native feature/semantic-dedupe-ledger-mode work:

1. semantic shred dedupe and downstream uniqueness guarantees in sof-observer
2. default lightweight duplicate handling with opt-in ledger-backed duplicate-shred tooling in the vendored gossip backend
3. cleaner bundled-gossip runtime handoff and shutdown behavior

Changes

• crates/sof-observer/src/app/state/dedupe.rs

  • Replaced the old double-storage shred dedupe approach with a shared semantic shred registry keyed by canonical shred identity.
  • Added ingress vs canonical stage markers so SOF still drops duplicates early and still guarantees downstream uniqueness.
  • Distinguished duplicate vs conflict observations based on signature mismatch for the same semantic shred identity.

• crates/sof-observer/src/app/runtime/runloop/driver.rs

  • Routed both ingress and canonical shred acceptance through the shared semantic dedupe state.
  • Preserved the contract that duplicate/conflicting shreds do not leak through to bot-facing downstream state/events.

• crates/sof-observer/src/runtime_metrics.rs

  • Added dedupe instrumentation for occupancy, queue depth, capacity vs expiry evictions, and ingress/canonical duplicate/conflict drops.

• crates/sof-observer/src/app/runtime/bootstrap/gossip/*

  • Cleaned up bundled-gossip runtime handoff and shutdown behavior.
  • Stopped treating repair/control-plane traffic as proof that the TVU ingest path is healthy.
  • Consolidated teardown through the shared gossip stop path instead of partial duplicate shutdown logic.

• crates/sof-solana-gossip/*

  • Made the ledger-backed duplicate-shred tooling opt-in instead of part of the default SOF path.
  • Default build now follows the lightweight SOF observer path and avoids pulling the heavier solana-ledger/RocksDB native dependency stack into the active graph by default.
  • Added feature-gated ledger mode for users who explicitly want the official ledger-backed duplicate-shred path.
  • Removed dead-code suppressions added during tuning and wired the actual queue/stats helpers into the running service.
  • Kept the stable legacy ingress path while removing dead experimental scaffolding.

• Docs

  • Updated root README, sof-observer README, and operations docs to describe SOF as low-latency market infrastructure.
  • Documented default duplicate handling vs opt-in ledger mode.

For slice-related changes, include:

• Affected slices
  • observer runtime/bootstrap/state
  • vendored gossip backend
• Cross-slice communication changes (if any) and why
  • sof-observer and sof-solana-gossip now have a cleaner contract around default lightweight duplicate handling vs opt-in ledger-backed duplicate-shred tooling.
• Migration requirements (if any)
  • none for existing observer users
  • users who specifically want ledger-backed duplicate-shred tooling must now opt in with the vendored gossip feature instead of getting that path by default

Motivation

Business motivation:

• SOF is being exercised under real HFT load, and duplicate downstream observations are fatal for strategy code.
• The default deployment should pay for low-latency live correctness, not the heavier validator-oriented duplicate-shred dependency stack that the observer path does not need by default.

Technical motivation:

• Eliminate duplicate/conflicting shred leakage before bot-facing downstream events.
• Reduce duplicated dedupe state while preserving both early-drop and final-canonical enforcement.
• Improve visibility into dedupe cache pressure and eviction behavior.
• Keep the vendored gossip backend lighter by default.

Alternative approaches considered:

• Keep the older double-cache dedupe design. Rejected because it duplicated state and tuning cost without adding correctness.
• Hard-remove ledger-backed duplicate-shred tooling from vendored gossip. Rejected because some users may still want the official heavier path.

Scope and impact

• Affected slices:
  • sof-observer runtime/bootstrap/state
  • sof-solana-gossip
• Data/API changes:
  • added semantic dedupe telemetry and stricter downstream uniqueness handling
  • added feature-gated ledger duplicate-shred mode in the vendored gossip crate
• Backward compatibility:
  • default SOF observer behavior remains lightweight and low-latency
  • ledger-backed duplicate-shred tooling is now explicit rather than implicit
• Performance impact:
  • lower duplicate-handling overhead through a shared semantic registry
  • better visibility into dedupe pressure for tuning
  • default vendored gossip build avoids unnecessary solana-ledger/RocksDB dependency cost unless that path is explicitly enabled
• Security impact:
  • no new privileged capability added
  • duplicate/conflict suppression is tightened before downstream event emission

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo make ci
cargo make clippy
cargo check --manifest-path crates/sof-solana-gossip/Cargo.toml --locked
cargo check --manifest-path crates/sof-solana-gossip/Cargo.toml --locked --features solana-ledger

All passed locally during branch validation for this work.

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
• Operations/runbook updates: docs/operations/advanced-env.md, README.md, crates/sof-observer/README.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

Trade-offs, follow-up work, or deviations from standards.

• The default SOF duplicate path is intentionally optimized for live observer correctness and latency, not the optional ledger-backed duplicate-shred workflow.
• The vendored gossip crate still carries upstream-style arithmetic lint allowances; that cleanup was intentionally left out of this branch to avoid mixing vendor-wide lint churn into HFT validation work.
• The later verifying-key-cache / inbound-preverify follow-up is tracked separately and is not part of this PR description.