feat(runtime): add Linux UDP busy-poll controls — validate on VPS

[Lythaeon]

Description

Add explicit Linux UDP busy-poll controls to SOF's runtime setup, apply the same tuning to the vendored gossip bootstrap sockets, and add a repo-owned VPS comparison harness so receive-strategy changes can be validated against the real Hetzner host profile.

This matters because busy polling is one of the few host-side levers available on VPS deployments where NIC RSS and IRQ steering are limited. The branch keeps it opt-in and measured instead of making it an implicit default.

Changes

• crates/sof-observer/src/runtime.rs
  • Added RuntimeSetup::with_udp_busy_poll_us(...), with_udp_busy_poll_budget(...), and with_udp_prefer_busy_poll(...).
  • Added a regression test that serializes the new env overrides.
• crates/sof-observer/src/ingest/config.rs
  • Added env parsing for SOF_UDP_BUSY_POLL_US, SOF_UDP_BUSY_POLL_BUDGET, and SOF_UDP_PREFER_BUSY_POLL.
• crates/sof-observer/src/ingest/receiver/io.rs
  • Applied Linux SO_BUSY_POLL, SO_BUSY_POLL_BUDGET, and SO_PREFER_BUSY_POLL on SOF's direct UDP ingest sockets.
• crates/sof-solana-gossip/src/node.rs
  • Applied the same busy-poll options to observer-facing UDP sockets in the vendored gossip bootstrap path (gossip, tvu, repair, serve_repair, ancestor_hashes_requests, related QUIC sockets, optional alpenglow/vortexor receivers).
• scripts/soak/vps-busy-poll-compare.sh
  • Added a deterministic 5-minute baseline vs 5-minute busy-poll comparison harness that builds the current branch binary, uploads it to the VPS, runs both slices sequentially, and prints the last queue/freshness counters.
• Makefile.toml
  • Added cargo make vps-busy-poll-compare.
• Docs / ops updates:
  • docs/operations/advanced-env.md
  • docs/gitbook/operations/knob-registry.md
  • docs/gitbook/operations/tuning-and-env.md
  • crates/sof-observer/README.md
  • docs/architecture/ard/0006-performance-and-hot-path-playbook.md
  • CONTRIBUTING.md

Affected slices:

• ingest
• app/runtime
• bundled gossip bootstrap integration
• ops/perf workflow docs

Cross-slice communication changes:

• none; this is socket tuning plus setup/env plumbing

Migration requirements:

• none; the feature is off by default

Motivation

Business motivation:

• Keep SOF viable on lower-control host classes where teams cannot rely on hardware RSS / IRQ steering but still want lower interrupt jitter on Solana UDP receive paths.

Technical motivation:

• Busy-poll is a real receive-strategy option for Solana-scale packet volume, but it should be explicit and measured.
• The repo needed a repeatable VPS workflow so future receive-path changes are compared on live traffic instead of only synthetic benches.

Alternative approaches considered:

• Leave busy-poll as an undocumented manual sysctl/socket experiment outside SOF. Rejected because it would not be reproducible or reviewable.
• Enable busy-poll by default. Rejected because the CPU trade-off is host-specific and should remain explicit.
• Limit busy-poll to SOF's direct UDP receiver only. Rejected because the public bootstrap path also uses vendored gossip-bound UDP sockets.

Scope and impact

• Affected slices: ingest, app/runtime, vendored gossip bootstrap path, operations docs
• Data/API changes:
  • new optional env/setup knobs:
    • SOF_UDP_BUSY_POLL_US
    • SOF_UDP_BUSY_POLL_BUDGET
    • SOF_UDP_PREFER_BUSY_POLL
• Backward compatibility:
  • preserved; defaults remain unchanged and busy-poll stays off unless configured
• Performance impact:
  • opt-in only
  • validated on the Hetzner VPS profile with a 5-minute baseline and a 5-minute busy-poll slice
  • result on this host was mixed rather than dramatic:
    • baseline: latest_shred_age_ms=5, latest_dataset_age_ms=8, receiver channel_len=107, verify max_len=3, output max_len=7, all drops 0
    • busy-poll: latest_shred_age_ms=2, latest_dataset_age_ms=15, receiver channel_len=120, verify max_len=4, output max_len=3, all drops 0
  • interpretation: busy-poll reduced shred freshness lag a bit on this host, did not improve every metric, and kept the runtime healthy with zero queue drops
• Security impact:
  • no new network listener or privilege surface
  • only socket option changes on Linux when explicitly requested

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo make ci
cd docs/gitbook && npm run build
SOF_COMPARE_RUN_SECS=300 bash scripts/soak/vps-busy-poll-compare.sh

VPS validation host:

• Hetzner ccx23, Ubuntu 24.04

VPS observations:

• baseline and busy-poll slices both stayed healthy
• both slices shut down cleanly via SIGTERM
• both slices had:
  • ingest_dropped_packets=0
  • dataset_queue_drops=0
  • packet_worker_dropped_packets=0
  • receiver_dropped_packets=0
  • verify_dropped_packets=0
  • output_dropped_packets=0
• busy-poll log confirmed both application points were active:
  • sof::ingest::receiver::core::io: configured UDP busy-poll socket options
  • solana_gossip::node: configured Linux UDP busy-poll socket options for observer-facing gossip sockets

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
  • docs/architecture/ard/0006-performance-and-hot-path-playbook.md
• Operations/runbook updates:
  • docs/operations/advanced-env.md
  • docs/gitbook/operations/knob-registry.md
  • docs/gitbook/operations/tuning-and-env.md
  • CONTRIBUTING.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

• This branch does not claim that busy-poll is universally better. On the validated VPS it behaved as a host-class tuning option, not a blanket improvement.
• The new scripts/soak/vps-busy-poll-compare.sh harness is intended to be reused whenever the UDP receive path changes again.