Git performance problems with large device count

[jerji]

Dear Maintainer,

My company uses Oxidized to store device configurations for various kinds of network devices. One of our instances currently has about 700 devices and the bare git repo has grown to 300~GB. Currently, it times out when requesting versions for devices probably because of the size of the repo.

Are there any optimizations you think might help in this situation or is there any profiling I could help with? Oxidized is an important tool for us and want to help where we can. :)

[robertcheramy]

Could you try running git gc on your repository?

[jerji]

I started it over night, we will see when it completes. Last tried to run it the server ran out of disk space while it was completing. I have since copied it to a new drive.

[ytti]

I recall looking into this years ago, trying to figure out if I'm using rugged incorrectly, leaving behind trash or if rugged is doing something incorrectly leaving behind trash or if this is expected accumulation of trash in git, I couldn't answer my question.

[robertcheramy]

git gc is not supported in libgit2/rugged: libgit2/libgit2#3247 so we have to document how to run it manually, this is what I want to document in docs/Troubleshooting.md.

I've read in #1805 (comment) that the git implementation in oxidized (or rugged?) is not the usual way, I'll have a look at it. As you already checked this, I'm not sure I will find something.

[ytti]

I don't think there is anything particularly odd in bare repo, we simply have no use for having checked out files laying about. We look at the file in the HEAD, and if it is same as one we fetched, we don't make commit, otherwise we make commit.

But as we're not manipulating any files, we don't really need to check them out. If we did check them out, we'd give more opportunities for people to break things.

[robertcheramy]

I've looked deeper into the output/git.rb.

I find it confusing that we write an index to a bare repository.

The README.md of libgit2/rugged reads every time the tree from the index, and does not write the index to disk.

This is a problem when someone tries to fix the git repository as in #1805 inside a clone and push it back: as it is a bare repository, git does not update the index, and oxidized ignores the current HEAD and works on the old index. I will address this in #1805 as it is off-topic here.

Beside the index subject , I can't see an incorrect use of rugged. We are simply making a lot of commits in a lot of files, and this slows git down after some time. The solution is to pack the loose objects into a packfile, which is git gcfor, and is (currently) not supported by libgit2/rugged. So the only current solution is to document how to run git gc.