Poor scalability for huge "flat" repositories

[borman]

• Gogs Version: master (0.5.8.1125 Beta)
• Bug Description:
  1. Create a new repository
  2. Push from a gargantuan repository with ~10k top-level directories and ~100k commits
  3. Navigate to your newly-populated repository
  4. See gogs turn your cpu into a frying pan

Turns out, gogs issues a bunch of subprocess calls like git log -1 --pretty=format:%H 713f036503b783df70c81e7174387af214ddcbd1 -- subdir-703 for each (top-level) directory item, which means at least 10k subprocess invocations per request. Good thing, it does not bring your system down with a fork bomb, nor it DoSes the whole gogs instance.

• How to reproduce:
  The following script may be used to create a model repository:

#!/bin/bash

ndirs=1000
nrounds=13

git init .

for i in $(seq 1 $ndirs); do
    mkdir subdir-$i
done

for round in $(seq 1 $nrounds); do
    for i in $(seq 1 $ndirs); do
        head -c 50 /dev/urandom | base64 > subdir-$i/data
        git add subdir-$i/data
        git commit -m"modify $i at round $round"
    done
done

On my laptop, it was enough to experience unacceptable request processing time:

[Macaron] Started GET /borman/bigrepo for 127.0.0.1
[Macaron] Completed /borman/bigrepo 200 OK in 45.588725781s
[Macaron] Started GET /borman/bigrepo for 127.0.0.1
[Macaron] Completed /borman/bigrepo 200 OK in 46.089607608s
[Macaron] Started GET /css/github.min.css for 127.0.0.1

[unknwon]

Thanks your feedback!

But I think it's too early to make Gogs perfectly handle huge repos, it involves cache technics which Gogs hasn't had right now but in the plan.

If you any ideas on it, please post as reply, then we can have some discussions.

[borman]

Seems like GitLab has the same issue, nothing special there:
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/controllers/projects/refs_controller.rb#L48
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/repository.rb#L237

I also wanted to check Atlassian Stash source, but they only provide it to paid customers.

How to solve the problem:

1. Are there any significant reasons (apart from simplicity) why git2go is not used as a git backend? This is actually not a solution, just a way to reduce git overhead. You could reduce IO complexity from Nfiles * SCAN to a single SCAN where SCAN is complexity of iterating through all commits. CPU complexity stays the same: O(Nfiles * Ncommits) (for each commit, you compare its tree with its parents' trees and mark entries that differ as updated, O(Nfiles * Nparents), but you could assume Nparents be equal to 1).
   I'd like to try and port Gogs to git2go since it looks like a nice entry-level project. As for the benefits, in the future it would allow to use git object store backends other than plain git packfiles, thanks to libgit2 extensible architecture.
2. As for caching, you need to cache (repository, path, commit_sha) -> last_modified_commit_sha mappings. And most of the time, you don't need commit_shas other than master. Such caching might also be needed for other cross-commit queries.

[unknwon]

Are there any significant reasons (apart from simplicity) why git2go is not used as a git backend?

Answer this a thousands times... I guess should put in FAQs, Go wrapper isn't as good as it looks, and it uses CGO which we are not comfortable with.

[unknwon]

Please follow on #1505