perf(gossip): preverify inbound CRDS and cache verifying keys

[Lythaeon]

Description

This PR tightens the vendored gossip receive path used by SOF's observer runtime.

The main changes are:

1. preverify and trim inbound CRDS payloads earlier, before paying full signature verification cost on obviously stale or irrelevant values
2. replace the explicit mutex-wrapped verifying-key LRU with a concurrent sharded cache better suited for heavy parallel packet verification
3. add telemetry so the new preverify and cache behavior can be tuned under real HFT/observer load

This is intended to keep the gossip ingest path cheaper under sustained packet volume without changing the higher-level runtime behavior that is already being validated on the live system.

Changes

• crates/sof-solana-gossip/src/cluster_info.rs

  • Added early preverify filtering for PullResponse and PushMessage CRDS values before full signature verification.
  • Reused batch-level now, CRDS timeout, and CRDS snapshot state instead of recalculating those per packet.
  • Wired a verifying-key cache into protocol verification.
  • Added env-configured cache sizing knobs:
    • SOF_GOSSIP_VERIFY_PUBKEY_CACHE_CAPACITY
    • SOF_GOSSIP_VERIFY_PUBKEY_CACHE_SHARDS

• crates/sof-solana-gossip/src/crds_value.rs

  • Added cached verification support for CrdsValue and batched verification helpers.
  • Replaced Mutex<LruCache<...>> with a sharded concurrent DashMap cache for expanded Ed25519 verifying keys.
  • Kept the cache bounded with approximate eviction per shard.
  • Added counters for cache hits, misses, insert failures, and evictions.

• crates/sof-solana-gossip/src/protocol.rs

  • Switched protocol verification to the cache-aware verification path for pull requests, pull responses, and push messages.

• crates/sof-solana-gossip/src/cluster_info_metrics.rs

  • Added metrics for preverify-dropped pull-response values and push-message wallclock drops.
  • Added metrics for verifying-key cache hits, misses, insert failures, and evictions.

• crates/sof-solana-gossip/src/gossip_service.rs

  • Wired the verifying-key cache telemetry into periodic gossip stats submission.

• crates/sof-solana-gossip/Cargo.toml

  • Added direct dependencies for ed25519-dalek and dashmap required by the new cache-aware verification path.

For slice-related changes, include:

• Affected slices
  • vendored gossip backend only
• Cross-slice communication changes (if any) and why
  • none at the SOF runtime API level; this is an internal receive/verify-path optimization and telemetry expansion inside sof-solana-gossip
• Migration requirements (if any)
  • none; runtime behavior is compatible, with new optional tuning knobs only

Motivation

Business motivation:

• SOF is being profiled under real HFT load, and gossip receive overhead matters directly to downstream timeliness.
• We need more visibility into whether gossip verification is spending time on values that can be rejected earlier.

Technical motivation:

• avoid paying full signature verification cost for CRDS values that can be rejected cheaply by local timeout / wallclock / retain filters
• avoid an explicit mutex hotspot on the verifying-key cache under parallel packet verification
• make cache behavior observable so sizing can be tuned with real workload data

Alternative approaches considered:

• keep the mutex-wrapped LRU cache. Rejected because it adds an explicit serialized hotspot on the same path we are trying to parallelize.
• remove caching entirely. Rejected because expanded public-key reuse is a real win on repeated gossip senders.
• preserve exact LRU semantics with a more complex concurrent cache. Rejected for now because the extra complexity is not justified; bounded approximate eviction is sufficient for this key-cache use case.

Scope and impact

• Affected slices:
  • sof-solana-gossip
• Data/API changes:
  • added two optional gossip env knobs for verifying-key cache sizing
  • added new gossip datapoint counters for preverify drops and cache behavior
• Backward compatibility:
  • backward compatible for existing SOF consumers
  • no required config change
• Performance impact:
  • less work spent verifying obviously stale/irrelevant gossip payload values
  • reduced explicit lock contention on the verifying-key cache path
  • better visibility into cache effectiveness and churn
• Security impact:
  • no new network capability or privileged behavior
  • signature verification remains enforced; this only adds earlier rejection and key reuse

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo make clippy
cargo make ci

Both passed locally on this branch.

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
• Operations/runbook updates:
  • none required for this internal vendored-gossip tuning pass

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

Trade-offs, follow-up work, or deviations from standards.

• The verifying-key cache is no longer exact LRU. It is now a bounded concurrent cache with approximate eviction, which is the right trade for the parallel verify path.
• The new telemetry should be used to confirm whether cache capacity/shard defaults are appropriate under sustained live load before widening this pattern elsewhere.