Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: fix faulty bin-image job #3040

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

crazy-max
Copy link
Member

relates to https://github.com/docker/buildx/actions/runs/13643604134/job/38138677210

System.IO.IOException: No space left on device : '/home/runner/runners/2.322.0/_diag/Worker_20250304-005208-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.322.0/_diag/Worker_20250304-005208-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.322.0/_diag/Worker_20250304-005208-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.Tracing.Dispose(Boolean disposing)
   at GitHub.Runner.Common.Tracing.Dispose()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)

image

We have no space left on runner for bin-image job since #2901

As quick fix we can free up disk space on the runner but would be better to distribute build in the future: #3005

@crazy-max
Copy link
Member Author

crazy-max commented Mar 4, 2025

Ah brownout of ubuntu-20.04 actions/runner-images#11101

image

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max crazy-max force-pushed the ci-fix-no-space-left branch from 99ea8c5 to 4636c80 Compare March 4, 2025 23:47
@tonistiigi
Copy link
Member

So what happened? I see it in other PR as well. Are we really using too much space (what changed) or did something happen with the runners?

@crazy-max
Copy link
Member Author

crazy-max commented Mar 5, 2025

Are we really using too much space (what changed) or did something happen with the runners?

It's related to adding netbsd builds in #2901, we were close to the limit I suppose. And we have not detected this issue until it was merged because we are akipping this job on PR as it takes quite some time. With #3005 it would be enabled on PR.

@tonistiigi
Copy link
Member

I don't understand how building small cli binary could run out of space. I did see this fail in PR as well but I've lost the link.

@crazy-max
Copy link
Member Author

crazy-max commented Mar 6, 2025

I don't understand how building small cli binary could run out of space.

Added repro in last commit temporarily: 69e83b7

See https://github.com/docker/buildx/actions/runs/13699603058/job/38315229163

image

I did see this fail in PR as well but I've lost the link.

Not sure how it would fail on PR as we are skipping this job entirely for this event atm:

if: ${{ github.event_name != 'pull_request' && github.repository == 'docker/buildx' }}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants