Linux runners fail to stream logs to UI

[anttijman]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.11.0

Deployment Method

Other

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. install gha-runner-scale-set-controller-0.11.0 to an AKS
2. install gha-runner-scale-set-0.11.0 to an AKS using a ubuntu jammy base image with runner version 2.325.0
3. Run a workflow job and observe how logs are not streamed to the github UI during the job execution. Logs are however available after job completion.

Describe the bug

We were using gha-runner-scale-set-controller-0.10.0 & gha-runner-scale-set-0.10.0 since couple of months. Then, suddenly (first observed on 30th of May), workflow job logs from linux runner scale sets were not streamed to the UI anymore.
We maintain and build our own linux runner images and the image had been built and deployed on 21th of May.
This problem does not happen with windows runner scale sets nor with VM runners.

We then upgraded gha-runner-scale-set-controller-0.11.0 & gha-runner-scale-set-0.11.0 but the problem still remains.

Describe the expected behavior

Logs should be streamed in the UI during job execution

Additional Context

I dont really know whether this is a bug in ARC or github backend or something else.

Here's a snippet from worker diagnostic logs which I think is relevant to this problem: https://gist.github.com/anttijman/768ad478caa3dd604ec70bb9c02db3cb

Controller Logs

https://gist.github.com/anttijman/3cde6809f6722d82c50ebce56106cd74 (there were 80k lines and creating gist failed so I took only the first ~1000 lines)

Runner Pod Logs

https://gist.github.com/anttijman/8268737cde0883087ec15901d415ccd3

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.