Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build master disk over-utilization #594

Closed
jugglinmike opened this issue Aug 28, 2018 · 1 comment

Comments

@jugglinmike
Copy link
Collaborator

commented Aug 28, 2018

Starting on 2018-08-23 and continuing through the weekend that followed, no
results were uploaded to wpt.fyi. The cause of this was exhaustion of disk
space on the build master.

For a given revision of WPT, the build master collects test results in "chunks"
and saves each in a dedicated fille. Once all "chunks" have been collected,
they are combined and uploaded to wpt.fyi. If one or more of these segments
fails, the data from the successful segments is persisted. This allows human
operators to investigate the cause of failure and "rescue" the failed
collection attempt by manually triggering only the failing segment(s). Rescuing
in this way is not always appropriate, so some partial result sets are never
completed. These prior result sets slowly accumulated on disk and ultimately
prevented further collection.

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 28, 2018

This project has suffered from issues related to disk space in the past, though
not from this particular data source. To resolve it, I have configured the
system to periodically remove partial result sets that are older than 30 days
(it is highly unlikely that rescue will take place after this period). The
commit is now present in the master branch and deployed to production:

bbcca43

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.