Release SOF 0.13.0: provider streams, hardening, perf, and docs refresh

[Lythaeon]

Description

Expands SOF from a gossip-focused observer into a multi-ingress Solana runtime.

This branch turns SOF into a reusable runtime foundation for Solana services: raw shreds, trusted raw-shred providers, Yellowstone gRPC, LaserStream gRPC, websocket transactionSubscribe, and typed generic provider ingress now feed one bounded runtime and one plugin/derived-state model with aligned semantics.

It also ships the 0.13.0 release line:

• sof 0.13.0
• sof-tx 0.13.0
• sof-gossip-tuning 0.13.0
• patched sof-solana-gossip 3.1.8-sof.6

Changes

Ingress

• Added built-in processed provider ingress for:
  • Yellowstone gRPC
  • LaserStream gRPC
  • websocket transactionSubscribe
• Added ProviderStreamMode::Generic as the typed custom-provider ingress surface for richer processed-provider integrations.
• Added explicit raw-shred trust posture:
  • public_untrusted
  • trusted_raw_shred_provider
• Added explicit provider replay modes:
  • Live
  • Resume
  • FromSlot(n)

Runtime And Correctness

• Unified processed-provider ingress into the main SOF runtime.
• Added capability-policy enforcement for plugins and derived-state consumers.
• Added replay dedupe and semantic duplicate suppression for provider updates.
• Added typed provider health, readiness gating, degraded-state handling, structured provider-runtime errors, and explicit EOF handling for bounded generic producers.
• Added unified transaction commitment selectors at the plugin layer:
  • .at_commitment(..)
  • .only_at_commitment(..)
• Clarified and enforced the semantic boundary for sof-tx:
  • supported on raw-shred/gossip runtimes
  • supported on ProviderStreamMode::Generic when the producer supplies the needed control-plane surface
  • explicitly rejected at runtime/config validation time on built-in transaction-first processed-provider modes

Plugin System And Dispatch

• Tightened the plugin contract and dispatch model.
• Documented and exposed shared-queue/drop behavior more explicitly.
• Plugin dispatch queues are bounded and overflow is drop-new, not drop-oldest.
• Queue ownership is shared per host/lane rather than per plugin, and SOF does not currently guarantee per-plugin fairness under pressure.
• Added queue-depth and drop metrics for plugin dispatch lanes.
• Preserved the ingest-first model: queue pressure drops the arriving event instead of blocking ingest.

Performance

• Continued hot-path optimization work across raw-shred and provider paths.
• Reduced avoidable copies, allocations, signature reparsing, and unnecessary transaction work.
• Added provider-side fast paths for ignored serialized transactions.
• Tightened Yellowstone and LaserStream conversion paths.
• Preserved net-positive performance on validated ingest-critical paths while adding the new hardening and observability work.

Representative validated fixture results on this branch:

• websocket parse: 162560 us -> 133309 us
• provider serialized ignore path: 42422 us -> 23760 us
• provider serialized accept path: 405493 us -> 365437 us
• provider tx-kind classification: 34112 us -> 4487 us
• Yellowstone mixed conversion: 43677 us -> 39289 us
• LaserStream mixed conversion: 45471 us -> 44830 us

Performance claim scope:

• no regression observed on the validated ingest-critical release fixtures for the main provider/runtime hot paths on this branch
• this is not a claim about every possible mixed workload or host topology

Observability

• Expanded runtime observability with:
  • provider health/state metrics
  • provider capability-degraded metrics
  • plugin queue depth and drop counters
  • readiness/degraded-state surfaces that reflect provider/runtime status more accurately
• These metrics now let operators detect backpressure, event loss, and degraded provider behavior in real time.
• Root README, crate READMEs, and GitBook architecture/operations docs were refreshed to match the runtime actually shipped on this branch.

Security And Trust Boundaries

• Made the trust model explicit across raw shreds and processed providers.
• trusted_raw_shred_provider disables local shred verification by default.
• Misuse of trusted_raw_shred_provider can lead to invalid data ingestion.
• Built-in providers now fail fast earlier on unsupported or invalid startup conditions.

Motivation

Business Motivation

• Make SOF a front-door framework for building Solana services instead of a narrow gossip-only observer.
• Let teams reuse a hardened runtime substrate instead of rebuilding ingest, replay, queueing, health, and correctness boundaries per service.

Technical Motivation

• Unify raw shreds, trusted raw providers, and processed provider streams under one runtime model.
• Preserve or improve hot-path behavior while adding replay, reconnect, health, and capability guarantees.
• Make the trust/correctness tradeoffs explicit instead of implicit.

Scope And Impact

• Affected slices:
  • provider ingress/runtime
  • raw-shred runtime and verification
  • plugin/dispatch framework
  • derived-state/runtime observability
  • docs and operations guidance
  • sof-tx adapter/docs boundary
• Backward compatibility:
  • existing raw-shred/gossip usage remains supported
  • built-in processed providers are intentionally stricter about unsupported hook surfaces
• Replay/dedupe scope:
  • provider replay dedupe is runtime-wide for the active provider ingress before plugin and derived-state dispatch
  • it is not a per-plugin cache
  • SOF does not run raw-shred and provider-stream ingest together inside one observer runtime, so there is no cross-family replay dedupe boundary inside a single running SOF instance
• Security impact:
  • sharper trust-boundary framing
  • better startup validation and degraded-state visibility
• Operations impact:
  • more explicit readiness/degraded-state behavior
  • more useful queue and provider-health metrics

Risks

Main remaining risks and mitigations:

• capability policy misconfiguration
  • mitigation: stronger docs, examples, capability metrics, strict-mode enforcement where semantics cannot be met
• event dropping surprises under pressure
  • mitigation: explicit plugin contract docs and queue/drop metrics
• trusted raw-shred misuse
  • mitigation: stronger docs and runtime warnings; the mode is explicitly documented as verification-off by default
• websocket durability remains best-effort
  • mitigation: explicitly documented; SOF cannot provide stronger replay semantics than transactionSubscribe plus the matching HTTP backfill path

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks
• Security/correctness review

Commands/results:

cargo make ci

Representative perf validation command:

perf stat -e cycles,instructions,branches,branch-misses,cache-references,cache-misses \
  target/release/deps/sof-52cd9e00a01162a4 \
  --ignored --exact provider_stream::websocket::tests::websocket_transaction_parse_optimized_profile_fixture --nocapture

Related Documentation

• Root README rewritten as the front-facing project overview
• crates/sof-observer/README.md updated as the crate-facing runtime/API guide
• crates/sof-tx/README.md clarified as standalone tx SDK plus optional SOF adapter integration
• GitBook architecture/operations/use-sof docs refreshed to match the current runtime and deployment semantics

Reviewer Checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

One-Line Summary

Turns SOF from a fast gossip observer into a semantically consistent, replay-aware, multi-ingress Solana runtime with explicit performance and correctness tradeoffs.