Runners keep exiting with System.IO.Exception network exception.

[jterzis]

I am running gh runner in GKE from helm chart defined in https://github.com/actions.

Versions (listener, controller, runner):

• 0.9.3

My runners as of this morning start and shortly exit with code 1 and a network error, System.IO.Exception: Unable to read data from the data connection. Operation cancelled . It looks like a problem with the upstream GH backend at broker.actions.githubusercontent.com. Restarting the controller, listener has not helped.

[indy-singh]

Looks very similar to what we had very recently; #3892 I've been keeping an eye for similar reports. I'm convinced that upstream the GH BE is broken somehow. But I'm not sure how to get the GH BE team to notice.

[sebodnar]

we are experiencing same issue with self-hosted runners. tried version 2.235 and 2.234, same issue with broker.actions.githubusercontent.com.

Long runners pickup from runners list. Long workflow run. Workflow gets Queued due to System.IO.IOException issue.
[2025-06-11 11:26:16Z ERR BrokerServer] System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled

[ldevai]

Same here, tried upgrading to linux-arm64-2.325.0 but same issue, even in different servers.
It prevent new jobs from being picked up for longer, sometimes minutes, as others reported.

Copilot provided a couple troubleshooting scripts, but the host is accessible, so far I couldn't find any other solution.

[2025-06-25 10:51:13Z INFO JobNotification] Entering JobCompleted Notification
[2025-06-25 10:51:13Z INFO JobNotification] Entering EndMonitor
[2025-06-25 10:51:13Z INFO BrokerMessageListener] Received job status event. JobState: Online
[2025-06-25 10:51:13Z WARN GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=...&status=Busy&runnerVersion=2.325.0&os=Linux&architecture=ARM64&disableUpdate=false has been cancelled.
[2025-06-25 10:51:13Z ERR  BrokerServer] Catch exception during request
[2025-06-25 10:51:13Z ERR  BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
...
[2025-06-25 10:51:13Z ERR  BrokerServer] #####################################################
[2025-06-25 10:51:13Z ERR  BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled
[2025-06-25 10:51:13Z WARN BrokerServer] Back off 7.195 seconds before next retry. 4 attempt left.
[2025-06-25 10:51:13Z INFO BrokerMessageListener] Get messages has been cancelled using local token source. Continue to get messages with new status.