Panic in op_state_manager during connectivity test shutdown

[sanity]

Summary

While running the three-node connectivity test, the runtime repeatedly logs a panic as the nodes are shutting down:

thread 'tokio-runtime-worker' panicked at crates/core/src/node/op_state_manager.rs:408:128:
called `Result::unwrap()` on an `Err` value: SendError { .. }

This occurs on feat/event-log-aggregator without any local patches, so it predates the mesh fix in PR #2041.

Steps to Reproduce

1. Checkout feat/event-log-aggregator.
2. Run cargo test --test connectivity test_three_node_network_connectivity -- --nocapture.
3. Watch the test output as the nodes tear down; the panic appears after the success summary.

Expected Behavior

All runtime threads should exit cleanly without unwinding after the test completes.

Actual Behavior

Multiple tokio-runtime-worker threads panic in op_state_manager.rs when SendError is unwrapped during shutdown. The test still reports success, but the panic pollutes logs and masks genuine failures.

Environment

• Branch: feat/event-log-aggregator
• Command: cargo test --test connectivity test_three_node_network_connectivity -- --nocapture
• Toolchain: nightly per workspace metadata

Additional Context

This surfaced while fixing #2039 (PR #2041) and appears in both the base branch and the patched branch. The panic is likely tied to teardown ordering in OpStateManager—a defensive if let Err(e) path would prevent the panic, but a deeper look is warranted to ensure we’re not dropping inflight work.

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-bug (98% confidence)
• P-medium (80% confidence)
• A-networking (90% confidence)

Reasoning: The report describes a runtime panic (crash) during test teardown, which is a bug. It occurs in networking-related code (OpStateManager during connectivity tests), so A-networking applies; because it also affects test output/log cleanliness and developer workflow, A-developer-xp is relevant. The panic does not appear to block release or cause data loss and tests still pass, so P-medium is appropriate. The fix may be more than a trivial unwrap check—teardown ordering and inflight work need investigation—so E-medium is a reasonable effort estimate.

Previous labels: T-bug, A-networking

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[sanity]

I attempted to reproduce the panic described in this issue on both origin/main (d668853) and origin/feat/event-log-aggregator (e7ab1fb) using the original repro command (cargo test --test connectivity test_three_node_network_connectivity -- --nocapture). In both cases the test now completes cleanly; no panic appears in op_state_manager.rs, even though the code still contains the unwrap() in question.

To verify we weren't missing an older regression, I also replayed the exact commit where we first noticed the failure (8fdf4c9), and it likewise exits without panic. That suggests the shutdown race was quietly addressed somewhere in the recent aggregator stabilization work and no longer triggers.

Even if the panic has stopped occurring, the shutdown path still calls unwrap() on a SendError, so it's trivial to reintroduce the crash if ordering changes again. The safest path forward is to replace the unwrap() with guarded handling so the runtime never panics on teardown.

Plan:

1. Add defensive handling in event_loop_notifier.notifications_sender.send(...) so any SendError is logged (or ignored) instead of unwrapped.
2. Include a brief explanation in the PR that the panic is currently unreproducible but the guard prevents regressions.

If anyone still sees the panic on a different revision, please drop the exact commit hash and command here so we can revisit. Otherwise I’ll send a patch covering step (1) to close this out.