fix(ops): harden soak restart and recovery checks

[Lythaeon]

Description

This PR replaces the old shell-based public-host soak scripts with a local-only Rust harness that runs directly on the target host from a normal cloned repo checkout. It matters because the restart, shutdown, and derived-state recovery checks are now part of the normal runtime hardening path, and the previous shell orchestration was too brittle around bootstrap retries, stale process cleanup, and shared port usage.

Changes

Detailed list of what changed:

• public_host_soak.rs: added a Rust soak harness binary with four scenarios:
  • observer-restart-loop
  • derived-state-restart-check
  • derived-state-crash-recovery-check
  • busy-poll-compare
• Makefile.toml: updated the soak tasks to run the Rust harness locally via cargo run -p sof --bin public_host_soak -- <scenario>.
• CONTRIBUTING.md: updated the contributor workflow to describe the local-on-host soak flow instead of SSH transport knobs.
• Removed the shell soak scripts that the Rust harness supersedes:
  • scripts/soak/vps-common.sh
  • scripts/soak/vps-observer-restart-loop.sh
  • scripts/soak/vps-derived-state-restart-check.sh
  • scripts/soak/vps-derived-state-crash-recovery-check.sh
  • scripts/soak/vps-busy-poll-compare.sh

For slice-related changes, include:

• Affected slices: operations / soak validation workflow, observer runtime restart validation, derived-state restart and crash-recovery validation.
• Cross-slice communication changes (if any) and why: the soak logic now executes through typed runtime/process/assertion code inside sof instead of shell parsing, but it does not change production runtime behavior.
• Migration requirements (if any): contributors now run the soak checks from a cloned repo on the target host; no SSH transport configuration is required by the harness.

Motivation

Business motivation:

• SOF is now close enough to serious adoption that restart, shutdown, and recovery validation needs to be repeatable and contributor-friendly on a real public host.
• A local Rust harness is easier to reuse and extend than VPS-specific shell scripts.

Technical motivation:

• the shell harness was brittle around bootstrap retries, shared UDP range serialization, and stale process cleanup
• typed Rust assertions are better than grep-based parsing for checkpoint progression, metrics health, and shutdown verification
• the soak checks should run the same way a contributor would run any other local test binary once the repo is already on the host

Alternative approaches considered:

• keep the shell scripts and only polish them: rejected because the assertions and orchestration were already getting too stateful and hard to maintain in shell
• embed SSH/transport logic into the Rust harness: rejected because the harness should be local-only on the target host
• move only the derived-state checks to Rust and keep observer restart in shell: rejected because all soak scenarios benefit from one typed harness and one lock model

Scope and impact

• Affected slices: soak/ops validation workflow, contribution workflow docs
• Data/API changes: new local-only soak binary public_host_soak; no production API changes
• Backward compatibility: shell soak scripts are removed and replaced by the Rust harness tasks
• Performance impact: no production runtime impact; only validation workflow changed
• Security impact: no new production listeners or permissions; the harness runs only when explicitly invoked on a host

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo fmt --all
cargo clippy -p sof --bin public_host_soak --all-features -- -D warnings

# on the cloned VPS repo
SOF_SOAK_CYCLES=1 SOF_SOAK_RUN_SECS=30 \
  cargo run -p sof --bin public_host_soak -- observer-restart-loop
# passed
# bootstrap completed
# ingest ok: latest_shred_age_ms=4 dedupe_entries=46708 dedupe_high_watermark=46708
# gossip queues ok: receiver_len=136 verify_max_len=11 output_max_len=28
# shutdown completed

SOF_DERIVED_STATE_RUN_SECS=20 \
  cargo run -p sof --bin public_host_soak -- derived-state-restart-check
# passed
# checkpoint advanced across restart: 226 -> 237

SOF_DERIVED_STATE_CRASH_RUN_SECS=20 \
  cargo run -p sof --bin public_host_soak -- derived-state-crash-recovery-check
# passed
# checkpoint advanced after recovery: 120 -> 148
# healthy_consumers=1
# fault_total=0
# replay failures stayed 0
# ingest and dataset drops stayed 0

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR: docs/architecture/ard/0006-performance-and-hot-path-playbook.md
• Operations/runbook updates: CONTRIBUTING.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

• The harness is intentionally local-only: it assumes the repo is already present on the host and does not contain transport setup.
• A shared .soak-lock directory serializes scenarios so restart/recovery checks cannot overlap on the same port range.
• Stale process cleanup now matches the full binary path, which avoids the old 15-character process-name limit that broke derived_state_slot_mirror cleanup on the VPS.