-
Notifications
You must be signed in to change notification settings - Fork 568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Buildkite: avoid unbounded disk space growth on runners #7139
Comments
See this thread on an unimplemented docker FR for ideas on how to implement docker gc. moby/moby#4237 |
I think the best option there is https://github.com/stepchowfun/docuum |
Saw another failure from low disk space: https://buildkite.com/iree/iree-samples/builds/96#de9a887f-7021-4276-b969-8eab5a2d2ab8
|
I've resized the disks to all be 1TB, so that should buy us some time |
bazelbuild/bazel#5139 (comment) would help with bazel disk caches if we want to use those in addition to a remote cache (note the "1" there could be substituted for some number of GB). |
Okay. This is also happening for the tiny VM we use for uploading pipeline configurations. The disk (10GB) is full there, which forbids it to uploading pipeline configurations, so we are seeing all tasks in the waiting queue.. I've created a new VM with 256GB storage for it for now. |
Huh, I'm curious what's eating all its space. I see you created a whole new VM, but note that you should be able to resize disks on the fly. Anyway, that VM had a stupid name, so fine that it went away. I think we should just delete it entirely? |
Whelp, the instance won't start up (likely due to being out of disk space), so I can't ssh into it to figure out what's going on. I've detached the disk and attached it to another instance so I can try to poke around and see what happened |
So some findings. When mounted elsewhere, the disk actually had 1.2G of free space. I don't really understand how that happens. I guess shutdown or detaching deleted some stuff ( gcmn@gcmn-test-instance:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 395M 5.3M 390M 2% /run
/dev/sda1 9.7G 1.4G 7.9G 15% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda15 124M 5.7M 119M 5% /boot/efi
tmpfs 395M 0 395M 0% /run/user/1001
/dev/sdb1 9.6G 8.4G 1.2G 89% /mnt/mako-uploader There are a few largeish directories: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/ | sort -rh | head -10
8.4G /mnt/mako-uploader/
3.1G /mnt/mako-uploader/usr
2.9G /mnt/mako-uploader/var
1.3G /mnt/mako-uploader/lib
1.1G /mnt/mako-uploader/home
222M /mnt/mako-uploader/boot
16M /mnt/mako-uploader/sbin
15M /mnt/mako-uploader/bin
6.9M /mnt/mako-uploader/etc
48K /mnt/mako-uploader/snap 2G of different gcp headers, for some reason: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/usr/src | sort -rh | head -10
2.2G /mnt/mako-uploader/usr/src
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1055
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1053
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1052
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1051
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1049
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1046
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1044
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1043
108M /mnt/mako-uploader/usr/src/linux-gcp-5.4-headers-5.4.0-1042 1.3G of "modules", whatever that is, that mirror the headers gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/lib/modules | sort -rh | head -10
1.3G /mnt/mako-uploader/lib/modules
251M /mnt/mako-uploader/lib/modules/5.4.0-1052-gcp
251M /mnt/mako-uploader/lib/modules/5.4.0-1051-gcp
250M /mnt/mako-uploader/lib/modules/5.4.0-1053-gcp
249M /mnt/mako-uploader/lib/modules/5.4.0-1055-gcp
238M /mnt/mako-uploader/lib/modules/5.4.0-1029-gcp
1.1M /mnt/mako-uploader/lib/modules/5.4.0-1049-gcp
1.1M /mnt/mako-uploader/lib/modules/5.4.0-1046-gcp
1.1M /mnt/mako-uploader/lib/modules/5.4.0-1044-gcp
1.1M /mnt/mako-uploader/lib/modules/5.4.0-1043-gcp A gigabyte of logs: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/var/log/journal/597226472db5e64da917fe51433dfb7c/ | sort -rh | head -10
977M /mnt/mako-uploader/var/log/journal/597226472db5e64da917fe51433dfb7c/ Half a gig of builds: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/ | sort -rh | head -10
574M /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/
288M /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/iree-android-arm64-v8a
138M /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/iree-build-configurations
78M /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/iree-samples
71M /mnt/mako-uploader/var/lib/buildkite-agent/builds/pipeline-uploader/iree/iree-benchmark A gig of snaps gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/var/lib/snapd | sort -rh | head -10
892M /mnt/mako-uploader/var/lib/snapd A gig because one time Hanhan did a bazel build on this machine: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /mnt/mako-uploader/home/hanchung/.cache | sort -rh | head -10
1.1G /mnt/mako-uploader/home/hanchung/.cache/bazel Let's contrast this with the boot disk from the new VM I just created: gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 / | sort -rh | head -10
1.3G /
1.1G /usr
208M /var
33M /boot
3.5M /etc
84K /home
44K /tmp
16K /lost+found
12K /root
4.0K /srv That's no gcp update headers or modules gcmn@gcmn-test-instance:~$ sudo du -xh --max-depth=1 /usr | sort -rh | head -10
1.1G /usr
799M /usr/lib
169M /usr/share
93M /usr/bin
17M /usr/sbin
52K /usr/local
20K /usr/include
4.0K /usr/src
4.0K /usr/libx32
4.0K /usr/lib64 So basically I think our Buildkite processes (IREE builds and logs) are responsible for 1.5G, there's some stuff that isn't growing, and the rest is GCP auto-updates, which hopefully at some point have garbage collection. Seems like a bigger disk is probably the answer in this case |
Thanks for looking into the details, @GMNGeoffrey! Didn't know that GCP auto update can be such resource consuming..
Oh I actually tried to fix the VM by SSH in and removed |
Is this issue obsolete with the migration to GitHub Actions? Do we have any similar problems there? |
Not as long as we have ephemeral runners |
We periodically run out of disk space in the buildkite build machines, which is annoying to deal with.
Right now each pipeline has its own checkout of the repository, which uses unnecessary disk space and adds checkout latency. https://buildkite.com/docs/agent/v3#experimental-features-git-mirrors would maybe fix this, but something custom might be required. IDK. This will matter more if we scale up the number of pipelines (which we may or may not do)
The text was updated successfully, but these errors were encountered: