PUT operations with auto-subscribe children don't deliver completion to client

[sanity]

Problem

In test_ping_blocked_peers_solution (currently #[ignore]), gateway PUT times out waiting for response even though the operation completes successfully on the node side.

Symptoms

From test logs:

• Gateway PUT starts, caches state, forwards (to self), seeds contract
• Child auto-subscribe spawned (transaction 01KAA6JC...) for parent PUT (01KAA6J...)
• Inbound RequestPut processed without errors
• Child subscribe created via start_subscription_request
• Session actor logs: "No clients waiting for transaction result: 01KAA6JC..." (expected - client waits on parent)
• Parent PUT never delivers result to WebSocket client
• wait_for_put_response times out

Root Cause Hypothesis

When child subscribe completes, parent PUT is not marked completed/delivered. The completion flow:

1. handle_op_result should place parent in root_ops_awaiting_sub_ops
2. OpManager::completed(child) should remove parent from awaiting
3. Spawn parent result delivery

Something in this flow is broken - either:

• Parent never makes it into root_ops_awaiting_sub_ops, OR
• Child completion doesn't trigger parent cleanup, OR
• Parent result delivery fails

Relevant Code

• crates/core/src/operations/put.rs - PUT operation handling
• crates/core/src/node/op_state_manager.rs - root_ops_awaiting_sub_ops tracking
• start_subscription_request - auto-subscribe creation
• OpManager::completed - child completion handling

Discovered During

Work on #2021 (blocked peers fix). This is a pre-existing bug - main branch fails the test even earlier (contract loading), but with that fixed, this PUT bug is exposed.

Testing

Run: cargo test test_ping_blocked_peers_solution -- --ignored --nocapture

Test is currently #[ignore] until this bug is fixed.

Impact

Blocks completion of #2021 test validation. May affect other PUT operations with child subscribes in production.

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-bug (98% confidence)
• P-high (86% confidence)
• E-medium (78% confidence)
• A-networking (88% confidence)

Reasoning: The report describes a functional failure where PUT operations with auto-subscribe children do not deliver completion to the client, which qualifies as a bug. It affects completion of tests and may affect production PUT flows, so priority is high. The fix requires moderate effort (investigating op state tracking and completion flow across OpManager/start_subscription_request), so E-medium. The problem appears rooted in the node's operation/subscription and message delivery flow (network/operation handling), so A-networking is appropriate; it also impacts testing/validation and developer workflow (failing ignored test and blocking related work), hence A-developer-xp.

Previous labels: none

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[netsirius]

After the change of 285ddd1 the put completes, but now the problem appears to be in the get + subscribe which is timing out before completing

[netsirius]

@sanity @iduartgomez

I've been debugging the test_ping_blocked_peers timeout and found that the WebSocket client uses mpsc::channel(1) for both request/response channels (stdlib/rust/src/client_api/regular.rs:107-108).

What I'm seeing:

When the test sends multiple UPDATEs without reading responses, it deadlocks:

• Test: 3x .send(UPDATE) → responses arrive → response_tx full (capacity 1)
• Handler: blocks on response_tx.send() waiting for test to .recv()
• Test: tries more .send() in next round → blocks because handler stopped reading request_rx
• Result: mutual deadlock, test never reaches the final get_all_ping_states() that would drain the buffer

Is this expected behavior? Should the API enforce strict send() → recv() interleaving, or is capacity 1 just overly restrictive?

What changes behavior:

Adding check_interval fixes it because it calls .recv() periodically between sends, preventing buffer saturation. But this feels like working around an API limitation rather than fixing root cause.

Should we increase channel capacity to support request pipelining, or is the current design intentional?

[iduartgomez]

In general if increasing channel capacity a bit helps with solving some potential deadlocks we can do that.

But we have to be mindful that if we need to increase capacity means there is a bottleneck somewhere in the system, and if there is a 1:1 relationship between producers and consumers there is something off that we probably need to take a look at.