perf(runtime): tighten inline tx visibility path — lower plugin latency

[Lythaeon]

Description

This PR is the full perf/gossip-ingest-hot-path branch landing.

It tightens SOF's transaction visibility path from shred ingest to plugin dispatch, makes inline transaction delivery explicit on the public API, adds the instrumentation and profiling tools used to validate the work, and refreshes the user-facing docs so the current runtime semantics are documented where users actually look.

Branch size against main:

• 61 files changed
• 11,620 insertions
• 1,277 deletions

The branch is intentionally squashed into 3 reviewable commits, but the actual change surface is broad and spans runtime, reassembly, FEC, dispatch, telemetry, examples, benches, profiling binaries, public docs, GitBook docs, and the companion gossip crate.

Changes

Detailed list of what changed:

1. Inline transaction visibility path

• crates/sof-observer/src/reassembly/inline.rs
  • introduced a dedicated inline reassembly path instead of treating inline tx visibility as a thin variation of deferred dataset processing
  • tracks chainable inline ranges/frontiers and supports retirement of already-emitted ranges
• crates/sof-observer/src/app/runtime/runloop/driver.rs
  • rewired the main runloop to use the dedicated inline path
  • tightened accepted-shred processing and inline advancement sequencing
  • ensures code-shred/FEC readiness can advance inline visibility without waiting for unrelated later events
• crates/sof-observer/src/reassembly/dataset/{core.rs,stream.rs,tests.rs}
  • updated dataset/open-prefix logic to coexist correctly with the dedicated inline path
  • expanded tests around anchors, chained boundaries, late prefixes, and open contiguous ranges

2. Transaction parsing and decode path

• crates/sof-observer/src/app/runtime/dataset/process.rs
  • replaced the earlier guessed tx-range approach with an entry-stream-prefix parser/cursor aligned to serialized Entry layout
  • split and measured sanitization/parse stages
  • removed unnecessary flattening/allocation on non-batch paths where validated
  • added matching regression coverage for prefix growth and tick+tx entry streams
• crates/sof-observer/src/app/runtime/dataset/mod.rs
  • re-export adjustments for the new parser/cursor surface

3. Packet-worker and runloop latency work

• crates/sof-observer/src/app/runtime/runloop/packet_workers.rs
  • packet workers now stream results during batch processing instead of holding visibility until the whole batch tail finishes
  • queue/result handling was tightened for lower hot-path withholding
• crates/sof-observer/src/app/runtime/runloop/driver.rs
  • reordered/deferred validated side work behind tx-bearing fast paths
  • reduced control-plane and relay-side interference on the main shred/tx path where accepted by measurement
• crates/sof-observer/src/app/runtime/dataset/dispatch.rs
  • split oversized completed-dataset worker bursts to reduce head-of-line latency spikes

4. FEC and shred-path improvements

• crates/sof-observer/src/shred/fec/core.rs
  • tightened FEC ingest/readiness accounting and reduced wasted work where validated
• crates/sof-observer/src/shred/fec/recover.rs
  • companion recovery-path adjustment
• crates/sof-observer/src/app/runtime/bootstrap/repair.rs
  • moved repair-side work off the tx-critical path where appropriate

5. Plugin framework and public API updates

• crates/sof-observer/src/framework/plugin.rs
  • made TransactionDispatchMode::Inline the explicit public contract
  • updated config helpers and trait docs to match the actual runtime behavior
• crates/sof-observer/src/framework/host.rs
• crates/sof-observer/src/framework/host/{builder.rs,core.rs,dispatch.rs,tests.rs}
  • host/dispatcher now explicitly support mixed inline and deferred transaction subscribers under one runtime
  • added/updated tests for explicit inline behavior and transaction classification/dispatch scopes
• crates/sof-observer/src/framework/events.rs
  • expanded public event docs and API-level examples
• crates/sof-observer/src/framework/mod.rs
  • export updates for the public framework surface

6. Runtime configuration and entrypoint changes

• crates/sof-observer/src/app/config/base.rs
• crates/sof-observer/src/app/runtime/entrypoints.rs
• crates/sof-observer/src/runtime.rs
  • added/documented opt-in runtime isolation controls
  • refreshed runtime composition/docs around the packaged observer runtime

7. Observability and measurement surface

• crates/sof-observer/src/runtime_metrics.rs
• crates/sof-observer/src/app/runtime/observability.rs
  • added tx-latency/source-path telemetry used to separate early inline dispatch from completed-dataset fallback
  • expanded runtime metrics for the new inline/FEC/dispatch behavior
• docs/operations/tx-latency-investigation-2026-03-23.md
  • captured investigation notes, measured results, and current conclusions

8. New profiling tools and benchmark coverage

• crates/sof-observer/src/bin/dataset_decode_profile.rs
• crates/sof-observer/src/bin/transaction_decode_profile.rs
• crates/sof-observer/src/bin/plugin_dispatch_profile.rs
• crates/sof-observer/src/bin/gossip_protocol_profile.rs
• crates/sof-observer/src/bin/gossip_vote_profile.rs
  • added standalone binaries for targeted measurement instead of relying on ad hoc local code changes
• crates/sof-observer/benches/hot_paths.rs
  • bench coverage updated for the hot paths touched by this branch

9. Examples and public-facing usage

• crates/sof-observer/examples/observer_inline_transactions.rs
• crates/sof-observer/examples/observer_inline_transactions_noop.rs
• crates/sof-observer/examples/observer_inline_transactions_with_dataset_noop.rs
• crates/sof-observer/examples/observer_transactions_noop.rs
• crates/sof-observer/examples/observer_transaction_batches_noop.rs
• crates/sof-observer/examples/observer_transaction_view_batches_noop.rs
• crates/sof-observer/examples/observer_multi_hook_noop.rs
  • updated/added noop and mixed-hook observers used for live validation and public usage reference

10. Docs.rs, README, GitBook, and operator docs

• crates/sof-observer/src/lib.rs
  • crate-root docs refreshed with public-facing examples
• crates/sof-observer/src/runtime.rs
• crates/sof-observer/src/framework/plugin.rs
• crates/sof-observer/src/framework/host/builder.rs
• crates/sof-observer/src/framework/host/core.rs
• crates/sof-observer/src/framework/events.rs
  • added item-level docs.rs examples directly on the public APIs users land on
• crates/sof-observer/README.md
• docs/architecture/framework-plugin-hooks.md
• docs/operations/advanced-env.md
• docs/gitbook/crates/sof.md
• docs/gitbook/use-sof/observer-service.md
• docs/gitbook/operations/tuning-and-env.md
• docs/gitbook/operations/knob-registry.md
  • refreshed inline semantics, tuning knobs, and public usage docs to match the current branch behavior

11. Companion gossip crate updates

• crates/sof-solana-gossip/src/{cluster_info.rs,crds_data.rs,crds_value.rs,duplicate_shred.rs,protocol.rs,weighted_shuffle.rs}
• crates/sof-solana-gossip/Cargo.toml
• crates/sof-solana-gossip/Cargo.lock
  • companion changes needed for the runtime/perf investigation and validation performed on this branch

12. Workspace and tooling metadata

• crates/sof-observer/Cargo.toml
• Cargo.lock
• Makefile.toml
  • dependency/tooling updates required by the new examples, profiling tools, and CI-validated branch state

For slice-related changes, include:

• Affected slices:
  • app/runtime
  • reassembly
  • shred
  • framework
  • runtime
  • docs
  • companion sof-solana-gossip
• Cross-slice communication changes:
  • inline tx delivery now flows through a dedicated inline reassembly path instead of piggybacking on deferred dataset-worker semantics
  • packet workers stream accepted results earlier into the main runloop
  • host dispatch explicitly separates inline and deferred tx subscribers while preserving one public plugin API surface
  • runtime metrics/observability now track the source path and timing of inline delivery
• Migration requirements:
  • downstream code should use TransactionDispatchMode::Inline
  • inline semantics/docs/examples now describe explicit inline delivery rather than a vague preference model

Motivation

Business motivation:

• improve tx visibility for HFT-sensitive observers without reducing the current data surface
• keep SOF competitive on latency while preserving correctness and delivery guarantees
• document the real runtime contract clearly enough that users can build against it safely

Technical motivation:

• reduce latency barriers between shred ingest and plugin dispatch
• separate inline tx visibility from deferred dataset reconstruction/worker semantics
• add first-class telemetry and tooling so hot-path changes are accepted or rejected using measured data
• make the public docs surface match the implementation, especially around inline delivery and tuning knobs

Alternative approaches considered:

• more dataset-worker tuning only: it helped earlier but stopped being the main bottleneck
• tx reconstruction directly from arbitrary raw shreds: rejected after source review because the protocol serializes Vec<Entry> before shredding
• hard CRDS/core partitioning on the reference 4-core VPS: tested and rejected as a default because it hurt overall host health on that machine

Scope and impact

• Affected slices:
  • app/runtime
  • reassembly
  • shred
  • framework
  • runtime
  • docs
  • sof-solana-gossip
• Data/API changes:
  • explicit TransactionDispatchMode::Inline semantics on the public plugin API
  • public docs.rs examples on runtime, plugin, host builder/host, and event APIs
  • new profiling binaries and refreshed examples/bench coverage
  • new documented runtime tuning knobs for opt-in isolation
• Backward compatibility:
  • standard/deferred observers remain supported
  • downstream code using older inline naming/expectations must update to the current explicit Inline API/semantics reflected in this branch
• Performance impact:
  • validated VPS A/B on the inline noop observer improved from 59.978 / 8.007 / 6.415 ms to 44.929 / 6.593 / 5.370 ms for first_shred / last_required_shred / ready -> plugin
  • accepted opt-in runtime isolation improved the software-controlled path further on the VPS (last=5.669 ms, ready=4.114 ms) without becoming the default topology
  • local hot-path fixtures and bench coverage improved on accepted changes; rejected experiments were not kept in the branch
• Security impact:
  • no new privileged interfaces were introduced; changes are within runtime/config/docs/tooling scope

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo test --doc -p sof
cargo make ci

Manual/perf validation:

• repeated VPS A/B runs on inline noop observers with exact latency counters
• local hot-path fixtures and bench coverage via crates/sof-observer/benches/hot_paths.rs
• targeted profile binaries for dataset decode, transaction decode, plugin dispatch, gossip protocol, and gossip vote paths

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
  • docs/architecture/ard/0003-slice-dependency-contracts.md
  • docs/architecture/ard/0007-infrastructure-composition-and-runtime-model.md
  • docs/architecture/ard/0008-observability-and-operability-standards.md
• Operations/runbook updates:
  • docs/operations/advanced-env.md
  • docs/operations/tx-latency-investigation-2026-03-23.md
  • docs/gitbook/operations/tuning-and-env.md
  • docs/gitbook/operations/knob-registry.md
  • docs/architecture/framework-plugin-hooks.md
  • crates/sof-observer/README.md
  • docs/gitbook/crates/sof.md
  • docs/gitbook/use-sof/observer-service.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

• This branch was intentionally squashed to keep review history manageable, but the actual code surface is broad. Reviewers should treat this as a branch-scale perf/runtime/docs landing, not a tiny targeted diff.
• On the 4-core reference VPS, hard CPU partitioning was not a good default. The opt-in runtime isolation knobs remain available, but the branch does not force that topology.
• The remaining dominant system-wide CPU cost on the reference VPS is still gossip CRDS verification. That work was measured, but broad CRDS topology changes were intentionally not made default in this branch because they hurt overall host health on the tested machine.