feat: add Jito transport and semantic gossip dedupe

[Lythaeon]

Description

This PR contains the full main...feature/semantic-dedupe-ledger-mode diff, which includes two strands of work because this branch was cut before the earlier sof-tx transport work landed on main:

1. add a typed Jito submit transport to sof-tx alongside the existing direct/RPC/hybrid modes
2. harden SOF's live gossip/observer path around semantic shred dedupe, lower-latency default duplicate handling, and cleaner vendored-gossip runtime wiring

The branch keeps SOF optimized for observer/HFT use by default while making the ledger-backed duplicate-shred tooling opt-in. The main benefit is that the default build no longer drags the heavier solana-ledger/RocksDB native dependency stack into the active dependency graph unless users explicitly enable that path.

Changes

• crates/sof-tx/src/submit/*

  • Added typed Jito transport support with SubmitMode::JitoOnly.
  • Added JitoJsonRpcTransport, typed endpoint/auth/config wrappers, and client wiring.
  • Kept defaults sensible for embedding: default mainnet block-engine endpoint, 10s timeout, no auth token, bundle_only = false.
  • Tightened UDP direct submit acceptance so a single transient UDP success no longer short-circuits success with no QUIC backing.

• crates/sof-tx/fuzz/*

  • Added standalone fuzz targets for submit mode behavior and core tx helpers.
  • Split targets into individual files for targeted fuzzing and easier CI/local runs.
  • Added fuzz coverage for:
    • submit_jito_mode
    • submit_rpc_only
    • submit_direct_only
    • submit_hybrid
    • routing_select_targets
    • signature_deduper
    • tx_builder_toggles

• crates/sof-observer/src/app/state/dedupe.rs

  • Replaced the old double-storage shred dedupe approach with a shared semantic shred registry keyed by canonical shred identity.
  • Added stage markers for ingress vs canonical enforcement so SOF still drops duplicates early and still guarantees downstream uniqueness.
  • Distinguished duplicate vs conflict observations based on signature mismatch for the same semantic shred identity.

• crates/sof-observer/src/app/runtime/runloop/driver.rs

  • Routed both ingress and canonical shred acceptance through the shared semantic dedupe state.
  • Preserved the low-latency contract that duplicate/conflicting shreds do not leak through to bot-facing downstream state/events.

• crates/sof-observer/src/runtime_metrics.rs

  • Added dedupe instrumentation for occupancy, queue depth, capacity vs expiry evictions, and ingress/canonical duplicate/conflict drops.
  • This is intended to make HFT/observer tuning visible instead of guessing at cache pressure.

• crates/sof-observer/src/app/runtime/bootstrap/gossip/*

  • Cleaned up runtime handoff and shutdown behavior for bundled gossip bootstrap.
  • Stopped treating repair/control-plane traffic as proof that the TVU ingest path is healthy.
  • Consolidated teardown through the shared gossip stop path instead of partial duplicate shutdown logic.

• crates/sof-solana-gossip/*

  • Made the ledger-backed duplicate-shred tooling opt-in instead of part of the default SOF path.
  • Default build now follows the lightweight SOF observer path and avoids pulling the heavier solana-ledger/RocksDB native dependency stack into the active graph by default.
  • Added feature-gated ledger mode so users who want the official ledger-backed duplicate-shred path can still enable it explicitly.
  • Removed dead-code suppressions added during tuning and wired the actual queue/stats helpers into the running service.
  • Kept the stable legacy ingress path while removing dead experimental scaffolding.

• Docs

  • Updated root README, sof-observer README, sof-tx README, docs index, and operations docs.
  • Documented SOF as low-latency market infrastructure rather than a generic observer framework.
  • Documented default duplicate handling vs opt-in ledger mode and the new Jito transport defaults/configuration.

For slice-related changes, include:

• Affected slices
  • observer runtime/bootstrap/state
  • vendored gossip backend
  • tx submit/routing surface
• Cross-slice communication changes (if any) and why
  • sof-observer and sof-solana-gossip now have a cleaner contract around default lightweight duplicate handling vs opt-in ledger-backed duplicate-shred tooling.
  • sof-tx adds a new submit transport path without changing existing direct/RPC/hybrid call sites.
• Migration requirements (if any)
  • No required migration for existing SOF observer users.
  • Users who specifically want ledger-backed duplicate-shred tooling must now opt in with the vendored gossip feature instead of getting that path by default.

Motivation

Business motivation:

• SOF is being exercised under real HFT load, and duplicate downstream observations are fatal for strategy code.
• The default deployment should pay for low-latency live correctness, not the heavier validator-oriented duplicate-shred dependency stack that the observer path does not need by default.
• sof-tx needed a first-class Jito path so transaction execution can be configured more granularly without out-of-band integrations.

Technical motivation:

• Eliminate duplicate/conflicting shred leakage before bot-facing downstream events.
• Reduce duplicated dedupe state while preserving both early-drop and final-canonical enforcement.
• Improve visibility into dedupe cache pressure and eviction behavior.
• Keep the vendored gossip backend lighter by default.
• Add fuzzable, typed, explicit transport surfaces in sof-tx.

Alternative approaches considered:

• Keep the older double-cache dedupe design. Rejected because it duplicated state and tuning cost without adding correctness.
• Hard-remove ledger-backed duplicate-shred tooling from vendored gossip. Rejected because some users may still want the official heavier path.
• Use stringly typed Jito configuration. Rejected in favor of typed endpoint/auth/config wrappers for better embedding UX.

Scope and impact

• Affected slices:
  • sof-observer runtime/bootstrap/state
  • sof-solana-gossip
  • sof-tx submit/fuzz/docs
• Data/API changes:
  • Added Jito submit mode and typed Jito transport/config API.
  • Added semantic dedupe telemetry and stricter downstream uniqueness handling.
  • Added feature-gated ledger duplicate-shred mode in the vendored gossip crate.
• Backward compatibility:
  • Existing direct/RPC/hybrid tx flows remain intact.
  • Default SOF observer behavior remains lightweight and low-latency.
  • Ledger-backed duplicate-shred tooling is now explicit rather than implicit.
• Performance impact:
  • Lower duplicate-handling overhead through a shared semantic registry.
  • Better dedupe pressure visibility for tuning.
  • Default vendored gossip build avoids unnecessary solana-ledger/RocksDB dependency cost unless that path is explicitly enabled.
  • Jito path adds a new submit option without regressing existing paths.
• Security impact:
  • No new privileged capability added.
  • Typed endpoint/auth configuration reduces accidental misconfiguration compared with raw string plumbing.

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo make ci
cargo clippy --workspace --all-targets --all-features --no-deps --locked -- -D warnings
cargo check --manifest-path crates/sof-solana-gossip/Cargo.toml --locked
cargo check --manifest-path crates/sof-solana-gossip/Cargo.toml --locked --features solana-ledger
cargo check --manifest-path crates/sof-tx/fuzz/Cargo.toml --bins

Additional validation on the tx branch work:

cargo test -p sof-tx --locked
cargo clippy -p sof-tx --all-features --all-targets --no-deps --locked -- -D warnings

Standalone fuzz targets were also run individually during branch validation for the sof-tx transport/helper surface.

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
• Operations/runbook updates: docs/operations/advanced-env.md, README.md, crates/sof-observer/README.md, crates/sof-tx/README.md, docs/README.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

Trade-offs, follow-up work, or deviations from standards.

• The vendored gossip crate still carries upstream-style arithmetic lint allowances; that cleanup was intentionally left out of this branch to avoid mixing vendor-wide lint churn into HFT validation work.
• The default SOF duplicate path is intentionally optimized for live observer correctness and latency, not the optional ledger-backed duplicate-shred workflow.
• If needed later, the ledger-backed duplicate-shred feature can be expanded without forcing RocksDB/native build cost back into the default SOF path.