Runner Fails to Connect to GitHub Actions Broker - TaskCanceledException / SocketException

[sebodnar]

Describe the bug
The self-hosted runner failed to complete a GET request to the GitHub Actions broker service. The operation was canceled due to a series of nested exceptions including TaskCanceledException, IOException, and SocketException (125): Operation canceled.

To Reproduce
when running workflows, workflow gets Queued and it takes long time to finish it due to error. Runner not changing status from Active to Idle and workflow is waiting when runner get Idle state.
BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.

Expected behavior
workflow run take normal time to run, workflows are not Queued. No error messages in Runner_20250611-***.log files

Runner Version and Platform

2.235, 2.234, actions-runner-linux-x64

OS of the machine running the runner? OSX/Windows/Linux/...
Ubuntu 22.04.3 LTS

Runner and Worker's Diagnostic Logs

[2025-06-11 09:09:50Z WARN GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=[edited]&status=Busy&runnerVersion=2.325.0&os=Linux&architecture=X64&disableUpdate=false has been cancelled. [2025-06-11 09:09:50Z ERR BrokerServer] Catch exception during request [2025-06-11 09:09:50Z ERR BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory1 buffer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpRequestMessage message, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable1 additionalHeaders, Uri requestUri, HttpContent content, IEnumerable1 queryParameters, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable1 sessionId, String runnerVersion, Nullable1 status, String os, String architecture, Nullable1 disableUpdate, CancellationToken cancellationToken) at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func1 func, CancellationToken cancellationToken, Int32 maxRetryAttemptsCount, Func2 shouldRetry) [2025-06-11 09:09:50Z ERR BrokerServer] ##################################################### [2025-06-11 09:09:50Z ERR BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory1 buffer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2025-06-11 09:09:50Z ERR BrokerServer] #####################################################
[2025-06-11 09:09:50Z ERR BrokerServer] System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (125): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory1 buffer, CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [2025-06-11 09:09:50Z ERR BrokerServer] ##################################################### [2025-06-11 09:09:50Z ERR BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled [2025-06-11 09:09:50Z WARN BrokerServer] Back off 6.934 seconds before next retry. 4 attempt left. [2025-06-11 09:09:50Z INFO BrokerMessageListener] Get messages has been cancelled using local token source. Continue to get messages with new status.

Thank you

[w007878]

We are encountering the same issue. Network requests cannot be forwarded to the host from the pod.

[dennispan]

Did you guys find the root cause or mitigation?

[sebodnar]

Did you guys find the root cause or mitigation?

Unfortunately no. Support suggest to check network connection into:
results-receiver.actions.githubusercontent.com *.blob.core.windows.net

but there all good in out side. still looking

[dennispan]

Thanks @sebodnar . Ever since we noticed this issue in actions runner pods, we are now seeing it in many other nodes on EKS as well. The symptom is the pod has no egress. Is that same for you guys?

[sebodnar]

@dennispan We're seeing an issue with GET request to https://broker.actions.githubusercontent.com on our self-hosted and self-hosted Dockerized runners. Every workflow run is affected by this.
You can check the Runner_20250605-1*-utc.log file on your side for details.

Example log output:
[2025-06-16 08:31:55Z ERR BrokerServer] Catch exception during request [2025-06-16 08:31:55Z ERR BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled

This results in workflows being unable to proceed to the next step, as the runner appears stuck in a Busy state.
Our architecture runs the entire workflow on a single runner (starting from the init state), so this issue is significantly increasing the runtime—from the usual 10–20 minutes to several hours. Fortunately, the workflows are still completing successfully despite the delay.

UPDATES
just to add. same issue I see on home K8s cluster and ARC(v.0.11.0)

[ldevai]

Same here, tried upgrading to linux-arm64-2.325.0 but same issue, even in different servers.
It prevent new jobs from being picked up for longer, sometimes minutes, as others reported.

Copilot provided a couple troubleshooting scripts, but the host is accessible, so far I couldn't find any other solution.

[2025-06-25 10:51:13Z INFO JobNotification] Entering JobCompleted Notification
[2025-06-25 10:51:13Z INFO JobNotification] Entering EndMonitor
[2025-06-25 10:51:13Z INFO BrokerMessageListener] Received job status event. JobState: Online
[2025-06-25 10:51:13Z WARN GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=...&status=Busy&runnerVersion=2.325.0&os=Linux&architecture=ARM64&disableUpdate=false has been cancelled.
[2025-06-25 10:51:13Z ERR  BrokerServer] Catch exception during request
[2025-06-25 10:51:13Z ERR  BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
...
[2025-06-25 10:51:13Z ERR  BrokerServer] #####################################################
[2025-06-25 10:51:13Z ERR  BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled
[2025-06-25 10:51:13Z WARN BrokerServer] Back off 7.195 seconds before next retry. 4 attempt left.
[2025-06-25 10:51:13Z INFO BrokerMessageListener] Get messages has been cancelled using local token source. Continue to get messages with new status.