Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Results collection interrupted due to disk capacity limits #629
After seven months of operation, the Buildbot database has grown beyond the 100 gigabytes we initially provisioned to store it. This prevented the build master from scheduling collection from Chrome and Firefox on 2018-11-11 and 2018-11-12.
I'd like to maintain that data for now, so I've doubled the Elastic Block Storage instance we use as a backing for the database. I then manually resized the partition via SSH:
(Although we typically prefer to express system configuration via Ansible, this operation does not represent an action that would need to be taken on a new deployment, so it's not useful to express it in terms of Ansible.)
When we began collecting results from Safari, we deployed an independent Buildbot "master" to manage the sole Mac Mini reserved for this purpose. Although that separation has limited the scope of failures, it has also bifurcated maintenance efforts.
In this case, I increased the disk space reserved for the original Buildbot master (see above), but I did not do the same for the MacOS-dedicated machine. Over the weekend, that machine reached disk capacity and became incapable of managing the Mac Mini worker.
The machine should be provisioned with dedicated EBS storage, and the current database should be transferred into place.
About a week ago, I provisioned additional storage as described above and manually mounted it. We have successfully collected results from both releases of Apple Safari every day since then without fail.