aarch64 builders: No space left on device

[Anachron]

Both the aarch64 builders run out of space now.

[the-maldridge]

Indeed it is. I'm trying to recover some space from chroots that didn't get cleaned up automatically. I'll see how much that gets me back and then go looking for more space if necessary.

[Anachron]

Thanks for the quick update, I wonder if we can also do something about a better monitoring/alerting which would prevent this from happening again?

[the-maldridge]

We have a prometheus server that monitors all the hosts, it just needs some alert rules to ping IRC or a system where we could handle outages. This has been on the todo list, but I haven't had time to setup alert rules yet. If you know how to write the rules, feel free to send me a PR.

You can play with the available data here: http://monitoring.voidlinux.org:9090/graph

[the-maldridge]

Got back almost a hundred GB. Restarting the failed builds now.