Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gitserver deletes all repositories if the disk size is ~= SRC_REPOS_DESIRED_FREE_GB #5059

Closed
ggilmore opened this issue Aug 2, 2019 · 7 comments · Fixed by #5106

Comments

@ggilmore
Copy link
Contributor

commented Aug 2, 2019

wantFreeG = env.Get("SRC_REPOS_DESIRED_FREE_GB", "10", "How many gigabytes of space to keep free on the disk with the repos")

Gitserver periodically runs a cleanup job so that it always keeps a certain amount of disk space free (default 10GB). However, if the actual disk size is ~= SRC_REPOS_DESIRED_FREE_GB, then gitsever will delete all of the repos on the instance as part of the cleanup process. This forces gitserver to reclone all those repos again, which results in bad/broken UX.

One way of fixing this is to make the cleanup threshold percentage-based instead of a fixed number.

cc @sourcegraph/core-services

@ggilmore ggilmore added this to the 3.7 milestone Aug 2, 2019

@ggilmore ggilmore changed the title gitserver deletes all repositories if the disk size is below 10gigs gitserver deletes all repositories if the disk size is below SRC_REPOS_DESIRED_FREE_GB Aug 2, 2019

@ggilmore ggilmore changed the title gitserver deletes all repositories if the disk size is below SRC_REPOS_DESIRED_FREE_GB gitserver deletes all repositories if the disk size is ~<= SRC_REPOS_DESIRED_FREE_GB Aug 2, 2019

@ggilmore ggilmore changed the title gitserver deletes all repositories if the disk size is ~<= SRC_REPOS_DESIRED_FREE_GB gitserver deletes all repositories if the disk size is ~= SRC_REPOS_DESIRED_FREE_GB Aug 2, 2019

@keegancsmith

This comment has been minimized.

Copy link
Member

commented Aug 2, 2019

So we should probably change this to instead target a certain percentage free (I propose 10%).

@keegancsmith

This comment has been minimized.

Copy link
Member

commented Aug 2, 2019

@ggilmore the customer who ran into this, have you given them a workaround? Also good find.

@juliasourceress

This comment has been minimized.

Copy link

commented Aug 2, 2019

@ggilmore

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2019

This happened with another customer today. Maybe we should look into adding a site-admin alert (telling you to increase your disk size) whenever gitserver runs this cleanup job?

@keegancsmith

This comment has been minimized.

Copy link
Member

commented Aug 6, 2019

We could surface this information in the admin progress notifications that @mrnugget added? Either way I think this is a bug, we should instead be targeting a percentage free.

@mrnugget

This comment has been minimized.

Copy link
Contributor

commented Aug 6, 2019

We could surface this information in the admin progress notifications that @mrnugget added?

We could create a status message when we're close to running out of disk space. It's harder to do a one-off message, as in "about to run the cleanup job", because that would require state.

@ijt

This comment has been minimized.

Copy link
Contributor

commented Aug 6, 2019

@keegancsmith, percentage free sounds good. I'll post a PR for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.