Reduce the time it takes for Octopus Server to determine a network connections to Tentacle is dropped.

[LukeButters]

The enhancement

The Need

Octopus Server and Tentacle establish long-lived TCP connections to transfer Requests and Responses. The timeouts for Reads and Writes over the TCP connection can be up to 10 minutes by default.

If the TCP connection between Tentacle and Server fails and is not closed cleanly by both parties (FIN or RST packets are not sent and received), it can take up to 10 minutes for Tentacle to detect the TCP connection has failed and to terminate a waiting Read or Write. We refer to a connection in this state as dropped.

This leads to undesirable delays when processing Requests and Responses. For example it can take 10 minutes for the Octopus Server to detect the dropped connection. Since 10 minutes exceeds the default RPC retry duration of 2.5 minutes, the communication to Tentacle will not be retried resulting in a failed Deployment, Runbook or Health Check.

Solution

Recently new lower TCP read/write timeouts were added to halibut. These settings enabling Octopus Server to detected a dropped connections in less time, allowing a OctopusServer to potentially recover from a dropped connection by retrying the RPC call.

This is available using an opt-in feature toggle since version 2024.1.3308

Links

• Adding feature toggle to use recommended timeouts and limits
• Reduce the time it takes for Tentacle to determine a network connection is dropped, by using Halibut's new recommended timeouts

[octoreleasebot]

Release Note: Improved detection of Network Errors when communicating with Tentacles, allowing Octopus Server to re-attempt communication.

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2024.1	2024.1.8781
2024.2+	all releases