My sourcegraph docker install has difficulty cloning my GitLab repos

[Tercan-Edmunds]

• Sourcegraph version: 2.12.3
• Platform information: Linux (4.18.16-arch1), Docker: 18.06.1-ce

Steps to reproduce:

1. Setup GitLab Repository in Site Config, with correct URL and API Token
2. Go to Respository List and enable one of the repos

Expected behavior:

• The repo is cloned and indexed

Actual behavior:

• The repo reports "Not Cloned" for a very long time (hours)

Log Example: (timestamps removed)

repo-updater | lvl=warn msg="Error updating repo metadata" repo=gitlab.vip.aws2/edmunds/consumers-inventory-elastic err="context canceled"
gitserver | lvl=warn msg="error cloning repo" repo=gitlab.vip.aws2/edmunds/consumers-inventory-elastic err="error cloning repo: repo gitlab.vip.aws2/edmunds/consumers-inventory-elastic (http://git:[TOKEN]@gitlab.vip.aws2/edmunds/consumers-inventory-elastic.git) not cloneable: signal: killed"

[nicksnyder]

@Tercan-Edmunds Sorry to hear that! Can you clarify if the repo ever clones? For example, is it slow or does it never work at all?

[Tercan-Edmunds]

A handful of repos will eventually clone (after a few hours), but the longer the queue, the longer the clone will take. In one install, with 100 repos, the last 10 had not been cloned after 72 hours...

[nicksnyder]

Thanks for the info.

The first log line comes from here:
https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/repo-updater/repoupdater/server.go#L116

This is an async request to update some metadata, and I suspect it is hitting some default timeout, so I don't think this log line is as interesting as the second one.

The second log line comes from here:
https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/gitserver/server/server.go#L763:59

We are trying to do a git ls-remote with a 5 second timeout and that appears to be timing out (e.g. signal: killed is what happens when the context times out).

From your own dev machine (or ideally from the machine that Sourcegraph is running on) can you run the following command:

time git ls-remote http://git:[TOKEN]@gitlab.vip.aws2/edmunds/consumers-inventory-elastic.git HEAD

If that takes close to or longer than 5 seconds we may just be hitting an aggressive timeout.

[Tercan-Edmunds]

Results in

git ls-remote 0.02s user 17.91s system 83% cpu 21.371 total.

(There are A LOT of tags due to our continuous integration system)

[Tercan-Edmunds]

Can the timeout be configurable?

[nicksnyder]

It is not currently possible to configure that timeout. Let me think about if there is a better command that we could run which would always be fast.

[nicksnyder]

@Tercan-Edmunds Does the command speed up if you provide the --heads tag?

time git ls-remote --heads http://git:[TOKEN]@gitlab.vip.aws2/edmunds/consumers-inventory-elastic.git HEAD

[Tercan-Edmunds]

actually, missed the HEAD in your original ls-remote command...
The total time bounces around 2.5s to 6 on the host.

In container --heads is significantly slower between 10-15s
In container no --heads runs between 5.5-9s

[nicksnyder]

Ah, looks like ls-remote does filtering locally...but there is a recent improvement in git 2.18 that might make this better.

The version of git in the Sourcegraph container should already be >=2.18 so can you try this:

# without --heads
time git -c protocol.version=2 ls-remote http://git:[TOKEN]@gitlab.vip.aws2/edmunds/consumers-inventory-elastic.git HEAD

# with --heads
time git -c protocol.version=2 ls-remote --heads http://git:[TOKEN]@gitlab.vip.aws2/edmunds/consumers-inventory-elastic.git HEAD

[Tercan-Edmunds]

In Container:
Without Heads [2.66,8.33,4.13,2.00,2.31] Avg: 3.87s
With Heads [3.64,3.22,3.75,9.02,11.43] Avg: 6.21s (and doesn't return anything like w/o --heads)