Cannot allocate memory when viewing repository

[floorish]

On v0.8.10 I get a 500 error whenever I want to open a repo with more than one file. I can access the other tabs (Issues/commits etc.), but the Code tab always results in an error.

[Macaron] Started GET /floorish/test for 77.*.*.*
2016/01/22 16:30:14 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/readme.md): fork/exec /usr/bin/git: cannot allocate memory
[Macaron] Completed /floorish/test 500 Internal Server Error in 88.956623ms

Just a simple test repository with a single readme.md works fine, but when I add another file I can't view the Code tab anymore.

Gogs works perfectly fine on v0.8.0 using the exact same config.

Actual memory usage looks fine as well:

[exmex]

Please test if it still happens in the latest 0.8.22 version.

[floorish]

There currently aren't any binaries available for that version right? I cannot compile it on the VPS:

$ go get -u -tags "sqlite" github.com/gogits/gogs
go build github.com/codegangsta/cli: /home/git/local/go/pkg/tool/linux_386/compile: fork/exec /home/git/local/go/pkg/tool/linux_386/compile: cannot allocate memory

And cross-compiling locally doesn't work either:

$ env GOOS=linux GOARCH=386 CGO_ENABLED=1 go build -v -tags "sqlite" 
# runtime/cgo
ld: unknown option: --build-id=none
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[unknwon]

@floorish how many files in the directory that fails?

[floorish]

When I add a second file to the repository, I can't open the code tab. When I remove the second file, commit and push with just 1 file it's fine.
So with 2+ files it fails on 0.8.10

[unknwon]

OK... how much memory do you have on VPS... I can't convert for bad math 😓

[floorish]

$ free -h shows 750MB, so not too much ;) (32bit Debian 7)

[unknwon]

750MB should be enough...

Gogs works perfectly fine on v0.8.0 using the exact same config.

Just to confirm you're sure about this?

[floorish]

100% looking at it right now with lots of files

[unknwon]

Thanks, got it...

[dakira]

I'm getting this from time to time, but it works after a reload. The log shows messages like these:

2016/01/29 01:26:17 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/tests): fork/exec /usr/bin/git: cannot allocate memory
2016/01/29 01:26:20 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/package.json): fork/exec /usr/bin/git: cannot allocate memory
2016/01/29 01:30:08 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/app): fork/exec /usr/bin/git: cannot allocate memory

There's more than enough memory available, though:

# free -m
             total       used       free     shared    buffers     cached
Mem:          8192       2347       5844        392          0       1455
-/+ buffers/cache:        892       7299
Swap:         4096          0       4096

[unknwon]

@dakira how many files in the directory that is having problem?

[dakira]

@unknwon It's the root of the repo. 12 files, 8 folders.

[unknwon]

@dakira thanks, can you help confirm that 0.8.0 works fine?

[dakira]

@unknwon I'm on 0.8.10. I'm not always getting this. When I hit reload all the time, I'm getting this in 1 out of 10 reloads.

If it helps I can set up a new installation with 0.8.0 and try with that.

[unknwon]

@dakira that would be best to try again 0.8.0, so we can be more sure about where the problem is.

[dakira]

Looks like I can't reproduce this with 0.8.0 while I can with 0.8.10 and 0.8.25 (by mercilessly hitting F5). Edit: I tried some more. This definately doesn't exist in 0.8.0.

[unknwon]

@dakira Got it, thank you!

[dakira]

Can confirm for develop, too.

[dakira]

This happens more often when there are more files and when Gogs has been running longer. It doesn't happen for a while after it has been restarted. Again with tons of memory available. Is there a workaround @unknwon ?

[unknwon]

@dakira Are you using precompiled binary or installed from source code?

[dakira]

As I mentioned, I can also reproduce this in develop, compiled from source code (as of 7 days ago).

[floorish]

Just to confirm, on 0.8.25 (precompiled, 386) I get the same error.

I installed Gogs 0.8.25 (precompiled, amd64) on a different vps, there it works fine. They're totally different setups though, so hard to compare unfortunately.

[dakira]

@unknwon as this is a regression, can I help somehow locating the problem if you can't reproduce yourself?

[ultimatedirty]

Same problem on 0.8.25.0129 (precompiled, amd64). Used Gogs Migration from freebsd/freebsd-ports and get those 500 errors in. Used Gogs Migration on gogits/gogs and its seems OK.

VPS: Ubuntu 14.04, 6 vCores, 8GB Ram

[dakira]

@ultimatedirty what exactly do you mean by Gogs Migration?

[ultimatedirty]

@dakira Top Menu Create -> New Migration

[ghost]

Thanks for answering :)
free -h shows:
total used free shared buffers cached
2.0G 225M 1.8G 68M 0B 165M

[unknwon]

@YugoCode thanks for the prove!

[NiklasRosenstein]

I just upgraded to 0.9.13.0318 and it seems to run well, though it feels very slow. When I go to the "Code" page to view the files on the master branch, the page loads in ~1800ms. However, when I go to view a single source file, it loads in ~400ms.

Is this difference in the page load time expected? Note that I do not know how Gogs actually gets the data from git. (Does it check out the whole working tree to get a list of the files?)

[unknwon]

@NiklasRosenstein #1518

[dakira]

This issue is back in 0.9.20.0404.

[Pastew]

Hi, I'm new to Gogs but I have the same error. I'm opening 10MB, one-commit newly pushed repo and I have 500 Internal Server Error in browser and in console:
[...routers/repo/view.go:136 Home()] [E] GetCommitsInfo: GetCommitByPath (/.idea): fork/exec /usr/bin/git: cannot allocate memory

Gogs version 0.9.13.0318

top output:

top - 14:45:28 up 49 min,  2 users,  load average: 0.04, 0.01, 0.00
Tasks:  40 total,   1 running,  39 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  1.1 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1048576 total,   308760 used,   739816 free,        0 buffers
KiB Swap:   524288 total,        0 used,   524288 free.   229948 cached Mem

[lucianolorenti]

I ran into the same problem with my server. Apparently it is related with the number of concurrent tasks performed in GetCommitsInfo of the Entries type: tree_entry.go..
I changed
taskChan := make(chan bool, 10)
to
taskChan := make(chan bool, 2)

and,for now, I can not cause the error again.

[bkcsoft]

@unknwon Maybe the amount of goroutines for this should be configurable? how hard would it be to do that ? (without passing it as an argument on every single call... 😒 )

[unknwon]

@bkcsoft yes, I think it is possible as a config option.

[y4my4my4m]

@unknwon
Getting this too in 0.9.28.0527 almost 50% of the time.
Only getting this error viewing the root of the repository (which is owned by an organization)

Log

2016/06/29 14:24:19 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/app.js): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:20 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/node_modules): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:24 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:29 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:46 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/public): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:52 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/package.json): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:54 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:26:20 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/public): fork/exec /usr/bin/git: cannot allocate memory

Memory:

              total        used        free      shared  buff/cache   available
Mem:           2.0G         44M        1.4G        162M        534M        1.7G
Swap:          4.0G          0B        4.0G

Any help?

I'm running on CentOS 7 and MySQL.

---

Edit:
taskChan := make(chan bool, 2)
seems to help, i was using centos7 packager, but now im building from scratch...just feels a bit unstable somehow.

Also noticed you've added
LimitMEMLOCK=infinity
LimitNOFILE=65535
To the gogs.service

Does this really work?

[unknwon]

@ubiko sorry, but really have no ideas how this problem is caused... maybe there are some limitation options we're not noticed because this is not happening to all machines (that's the hardest part for debugging).

[y4my4my4m]

@unknwon

taskChan := make(chan bool, 2) is fixing it...but does it mean it's slower than if it was set at 10?

[unknwon]

@ubiko yes, that's right...

[bkcsoft]

This is most likely related to `ulimit`, there's a couple of issues about that already, check them out 😃

[ghost]

@ubiko The line of code you posted is fixing the problem??? I don't care if it's slow, I need it to work at least. Where do I have to put the line? Or could you build a binary with this fixed line?

Thank you :)

[sjahreis]

@bkcsoft In my case it's not related to ulimit. No limitations configured...

Please add as config option!

[y4my4my4m]

@unknown no worries, it's a pretty hard issue to figure out also. I gotta thank you for making Gogs before I'd complain. 👍

@YugoCode Yes it is fixing it. It's more of a workaround than a fix though, credit goes to: @lucianolorenti (Higher up in the comments) . You can fix it by modifying tree_entry.go . You need to work from the source, you can't fix this if you're using a docker/packager. You need to build the app again.
Good luck.

[toudi]

I was also affected by this issue on the VPS. I am wondering whether maybe this has something to do with number of cores available? Because on my VPS I got this 500, but on my OrangePi gogs runs flawlessly and I don't have this error. The only difference seems to be in the architecture (VPS is i386 vs arm7 on the OrangePi) and the number of cores (OrangePi has 4 cores whereas my VPS has only one).

If this is false impression, please tell me and I'll remove my comment so that I don't introduce unnecesary mess.

Thank you so much for gogs!

[botboe]

As @sjahreis already mentioned, a config option would be awesome!

[botboe]

Something new here?

[y4my4my4m]

@toudi @Mad-Onion
I believe it has to do with available Swap memory on your VPS setup.
Either increase swap capacity on your VPS (if your host allows this) or your setup, you might be out of luck (as I was, decided to change environment)

[WolfgangKluge]

There's the pull request gogs/git-module#26 from 0xbaadf00d, but unfortunetely no progress since one month...

[dakira]

@WolfgangKluge This project seems to be dead. There is an extremely active fork here: https://github.com/go-gitea/gitea. I've been asked to wait for the 1.0 release before I switch over. But that should be soon.

[unknwon]

Please notify when this happens again.

[paradajz]

Happens to me as well. CentOS 6.8. Seemingly random behaviour. Latest Gogs. Let me know if you need more details.

[extraymond]

Happened in a docker container on google compute engine.
Hope this gets fixed, thx.