feat: RPC timeout exception and fix main's blocking on writing to sockets

[as51340]

What

Adds end-to-end RPC timeout detection and error propagation for storage transactions (data replication). When a replica becomes unresponsive during replication, the main instance now detects the timeout at the TCP/socket level and surfaces a distinct
TimeoutReplicationError through the storage commit path, ultimately throwing a ReplicationException with an actionable message to the user. Previously, all replication failures (timeout, connection reset, protocol error) were collapsed into a single generic error.

Additionally, introduces a configurable replication_mode parameter to the HA stress test infrastructure and a new rpc_timeout stress workload that uses iptables to simulate network partitions.

Also fixes a scenario in which main would block because sending data to replica would've been blocked. This is solved by introducing TCP_USER_TIMEOUT parameter.

Why

In HA clusters, when a SYNC or STRICT_SYNC replica goes down or becomes unreachable, the main instance's send() would block for minutes (default TCP retransmission timeout) before failing. Users had no way to distinguish a timeout from other replication failures,
making diagnosis difficult. With TCP_USER_TIMEOUT set on sockets, the kernel tears down the connection within seconds, and the distinct error type lets users understand the root cause immediately.

How

The change threads timeout information through four layers:

• Socket layer (src/io/network/socket.cpp): Adds SetUserTimeout() (TCP_USER_TIMEOUT) and WaitForReadyWrite() with timeout detection. Socket Write() now returns std::expected<void, ClientCommunicationError> distinguishing TIMEOUT_ERROR from
  GENERIC_ERROR.
• RPC layer (src/rpc/client.hpp, src/rpc/exceptions.hpp): Introduces RpcTimeoutException (sibling of GenericRpcFailedException, both inherit RpcFailedException). StreamHandler::SendAndWait()/SendAndWaitProgress() and the SLK write callback translate
  socket timeout errors into RpcTimeoutException. Adds InProgressRes to the generic RPC messages layer (moved from storage-specific rpc.hpp).
• Replication layer (src/storage/v2/replication/): ReplicationStorageClient::FinalizePrepareCommitPhase() and FinalizeTransactionReplication() catch RpcTimeoutException separately and return ClientCommunicationError::TIMEOUT_ERROR.
  TransactionReplication::ShipDeltas() returns std::expected<void, ClientCommunicationError> (was bool) and prioritizes timeout errors over generic errors when aggregating across replicas.
• Storage/interpreter layer (src/storage/v2/inmemory/storage.cpp, src/query/interpreter.cpp): InMemoryStorage::Commit() maps ClientCommunicationError::TIMEOUT_ERROR to TimeoutReplicationError{} in the StorageManipulationError variant. The interpreter's
  commit path checks for TimeoutReplicationError and throws ReplicationException(rpc::kRpcTimeoutMsg).

Other changes:

• Jepsen tests: Added main-reached-rpc-timeout? handler across HA bank, create, multi-tenancy, and replication test suites. Changed several error handlers from :ok to :info for correctness (these are indeterminate outcomes, not confirmed successes).
• Stress test infra: Added replication_mode input to stress_tests.yml / reusable_stress_tests.yaml, threaded through mgbuild.sh → deployment.sh. New rpc_timeout workload uses iptables to block a replica's replication port during a heavy write.
• StreamAndFinalizeDelta (system replication client in src/replication/include/replication/replication_client.hpp): Catches RpcFailedException (base class) instead of only GenericRpcFailedException to handle both timeout and generic failures gracefully.

Testing

• Unit tests updated: rpc_timeouts.cpp, rpc_in_progress.cpp, replication_rpc_progress.cpp, snapshot_rpc_progress.cpp — all timeout scenarios now expect RpcTimeoutException instead of GenericRpcFailedException.
• New stress test: tests/stress/ha/workloads/rpc_timeout/workload.py — spawns a heavy write query, blocks a replica's replication port via iptables, verifies that the query fails with a timeout or sync replication error. Enabled for native HA, disabled for
  Docker/EKS (requires root/iptables access).
• Jepsen tests: Updated all HA and replication suites to handle the new timeout error as an indeterminate outcome (:info)

[as51340]

Tracking

• [Link to Epic/Issue]

Standard development

• Update unit/E2E tests
• Compare the benchmarking results between the master branch this branch

CI Testing Labels

• Select the appropriate CI test labels (CI -build=build-name -test=test-suite)

Documentation checklist

• Add the documentation label
• Add the bug / feature label
• Add the milestone for which this feature is intended
  • If not known, set for a later milestone
• Write a release note, including added/changed clauses
  • When a SYNC or STRICT_SYNC replica becomes unresponsive during replication, the main instance now detects the network timeout within seconds (via TCP_USER_TIMEOUT) instead of blocking for minutes, and returns a distinct error message — "Main reached an RPC timeout
    while waiting for the response from at least one replica" — so users can immediately identify the root cause rather than seeing a generic replication failure. For SYNC clusters the transaction still commits on main; for STRICT_SYNC clusters the transaction is aborted
    on all instances, consistent with existing behavior. No configuration changes or user action is required — the timeout detection is automatic. #3942
  • What has changed? What does it mean for a user? What should a user do with it? [#{{PR_number}}]({{link to the PR}})
• [ Documentation PR link memgraph/documentation#XXXX ]
  • Is back linked to this development PR

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
8.5% Duplication on New Code

See analysis details on SonarQube Cloud