Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all old logs, not just the zipped up ones. #710

Closed
wants to merge 1 commit into from

Conversation

@edunham
Copy link
Contributor

edunham commented Aug 2, 2017

Fixes #506

The only stuff in these directories is the various logs put there by Buildbot, so we're safe to ditch them.

5 days might be a bit to aggressive but it's what we have before to no major ill effects so I left it. If we want partial logs for a month, it'd be just as easy to run a separate cron job to delete the non-zipped files a bit less enthusiastically.


This change is Reviewable

@edunham edunham force-pushed the edunham:oldlogs branch from 3b46863 to a307ac7 Aug 2, 2017
@edunham
Copy link
Contributor Author

edunham commented Aug 4, 2017

Travis has failed twice now on assorted intermittents unrelated to this change.

Copy link
Member

aneeshusa left a comment

I find that having build history is definitely useful; currently there is ~11GB of history (the disk is 63GB). I would want to keep more than 5 days of history.

Here's some approximate numbers I just looked up:
Other misc files (not logs or bz2 files) (via find /home/servo/buildbot/master -maxdepth 1 -type f | xargs du -sch): 391 MB

Current total bz2 archives (via find /home/servo/buildbot/master -name '*.bz2' -print0 | du -sch --files0-from=-): 4.1Gb

All files without bz2 files (via find /home/servo/buildbot/master -type f -mtime -60 -regextype grep -regex '.*[^b][^z][^2]$' -print0 | du -sch --files0-from=-, changing the number of days):
1 day - 204 MB
2 days - 221 MB
3 days - 235 MB
5 days - 269 MB
15 days - 365 MB
30 days - 486MB
60 days - 698 MB
90 days - 2.3 GB
120 days - 2.5 GB
150 days - 2.7 GB
180 days - 2.8 GB
365 days - 3.7 GB
All time (no logs before 508 days ago) - 6.1 GB

I think if we do get rid of logs, we could probably get away with a 6 month or 1 year cutoff.

@@ -90,7 +90,7 @@ buildbot-github-listener:
remove-old-build-logs:
cron.present:
- name: 'find {{ common.servo_home }}/buildbot/master/*/*.bz2 -mtime +5 -delete'
- name: 'find {{ common.servo_home }}/buildbot/master/*/* -mtime +5 -delete'

This comment has been minimized.

@aneeshusa

aneeshusa Aug 20, 2017

Member

I think this will cause an argument too long error, so we'll need to rewrite this differently to not expand all the files in the command line args.

This comment has been minimized.

@aneeshusa

aneeshusa Aug 20, 2017

Member

Getting rid of the second /* at the end should probably do the trick. For the bz2 cron, we can add a -name '*.bz2' as well, and make a separate one for all logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.