An error is occurred : too many open files

[chanyuenpang]

it shows 500 or 400, description is
An error is occurred : open xxx.git too many open files

[linquize]

What operation did you do? For example, the URL?

[unknwon]

Thanks your feedback!

But it is a common problem on system setting, and Gogs really can't help with it. 😳

For example: http://askubuntu.com/questions/181215/too-many-open-files-how-to-find-the-culprit

[simonszu]

Well, maybe it should? I have set the ulimit to 300k files - gogs was greedy and snatched them all. So i increased to 1 million open files, gogs was greedy and snatched them all. All while just browsing one repository with 500 commits and around 1000 files in sum. Regarding the fact that the default value is around 65k files, i would assume that there is something weird happening with gogs's file management, maybe it doesn't close them or something.
I could open these repostiorys with cgit and gitlab very well, with the default value of 65k max open files...

If you really say that there's nothing that gogs can do, what are good values to set the file limt to? 10 million? 100 million? A billion?

And - according to my CPU usage and the page load time - is it possible that gogs is crawling the repository recursively and trying to read each file at its own? Then it is some kind of gogs's own problem.

[unknwon]

@simonszu Currently Gogs only reads all files/directories of current browsing level.

If you want to dive into this problem more, you should provide more information about your Gogs version, system, etc. And where the error occurs.

[simonszu]

Okay, sure.
So, details to the affected repository: 8652 files, 492 commits. It is an /etc-Directory of one of my hosts, managed by etckeeper.
The host which i run gogs on is a VPS with Debian Jessie. 6GB RAM, Dualcore with 2.2GHz each. I am using gogs from binary, version 0.8.10.

So, i will startup gogs manually (usually via systemd) and check how many open files will appear there after each pageload. The hard- and soft limit is set to 1 million open files, system wide.

Beginning: gogs is stopped, there are around 9000 files open by the dropbox daemon i have running on the same host, and around 600 files by other processes. I have checked it via the lsof-awk-combination found on the page you linked above.

Starting gogs, but not accessed it via browser yet: gogs has 24 open files.
Acessing it, logging in, visiting my dashboard: 24 open files.
Visiting the specified repository: Number of open files increases to 304, gogs is, after dropbox, the process with the second most open file handlers on my system.
Back to dashboard, and opening the repository again: 452 open files by gogs. Why is this number higher than on the previous visit?
Browsing a commit and it's filetree-snapshot a week ago: 560 open files.
Pulling and pushing via git: 560 open files
Opening another repository in gogs: Loading time around 4 seconds, 608 open files afterwards.
Going back to dashboard, opening the first repository again: 744 open files.

So you see, once everytime i open a repository or a commit in a repository, the number of open files increases. And if you do some heavy-ish work on your repos (i am currently, because i am rebuilding my network monitoring setup), you will on some moment run against your open-files-limit.

At the moment i am unsure how i could get to the limit of 1 million open files yesterday after running gogs for 4 days, but i think you see the point.

[unknwon]

Thanks... but I still don't see the error message line..?

[simonszu]

Yeah. I had to use gogs until the error occured the next time. This is now. The error is:
Jan 05 14:25:43 tharan gogs[5630]: 2016/01/05 14:25:43 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/subuid-): pipe2: too many open files

which is strange, because of the ouptut of lsof, gogs has currently 408 open files.

I got this error when i browsed to a tree view of a certain commit (selecting a commit in the list of commits and clicking on the blue "Browse Source" in the upper right corner).

Please note that my limit of open files is still set to 1 million.

The only solution for me to resolve this problem for now is that i have to restart the gogs service.

[unknwon]

The error line seems outdated, can you upgrade to latest release then try again?

[simonszu]

Of course. I see on Github that you are in release 0.8.15, but the latest
binaries on gogs.io are 0.8.10. Where can I get the newest binaries?

On Tue, Jan 5, 2016, 21:45 无闻 notifications@github.com wrote:

The error line seems outdated, can you upgrade to latest release then try
again?

—
Reply to this email directly or view it on GitHub
#1270 (comment).

[unknwon]

@simonszu OK, never mind I can find it by viewing tag on GitHub.

So how many files in the directory you browser in reality? (the commit view you talked about, is it always happening for arbitrary commit?)

[unknwon]

And, is it possible that your max open file limit didn't apply to the context that runs Gogs (start Gogs as service)?

[winsx]

I have the same problem to use gogs browse a git repository that one directory have so many files,then it shows 500

• Gogs's version is 0.8.19.0111
• go's version is go1.6beta1 linux/amd64
• in fedora 23 use systemd start Gogs as service
• check log file and find infomation below

[git@linux log]$ tail gogs.log
2016/01/12 21:03:23 [I] Git Version: 2.5.0
2016/01/12 21:03:26 [I] SQLite3 Supported
2016/01/12 21:03:27 [I] Run Mode: Production
2016/01/12 21:03:31 [I] Listen: http://0.0.0.0:2048
2016/01/12 21:06:08 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//bowling.png): pipe2: too many open files
2016/01/12 21:27:21 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//blue_heart.png): pipe2: too many open files
2016/01/12 21:27:23 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//bullettrain_side.png): pipe2: too many open files
2016/01/12 21:27:33 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//yum.png): open /dev/null: too many open files
2016/01/12 21:27:36 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//busts_in_silhouette.png): pipe2: too many open files
2016/01/12 21:27:39 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-hdpi//beers.png): open /dev/null: too many open files

[git@linux gogs]$ cat /proc/sys/fs/file-max 
813396

[git@linux gogs]$ ulimit -Sn
65536

[git@linux gogs]$ ulimit -Hn
131072

I stop gogs as service and start gogs in a shell session then re-browse git repository that failure 500 above, it had success access but very slow.

[git@linux gogs]$ sudo systemctl stop gogs.service

[git@linux gogs]$ ./gogs web
2016/01/12 21:49:07 [T] Custom path: /home/git/opt/gogs/custom
2016/01/12 21:49:07 [T] Log path: /home/git/opt/gogs/log
2016/01/12 21:49:07 [I] Gogs 0.8.19.0111
2016/01/12 21:49:07 [I] Log Mode: File(Info)
2016/01/12 21:49:07 [I] Cache Service Enabled
2016/01/12 21:49:07 [I] Session Service Enabled
2016/01/12 21:49:07 [I] Git Version: 2.5.0
2016/01/12 21:49:07 [I] SQLite3 Supported
2016/01/12 21:49:07 [T] Doing: CheckRepoStats
2016/01/12 21:49:07 [I] Run Mode: Production
2016/01/12 21:49:07 [I] Listen: http://0.0.0.0:2048
[Macaron] Started GET /coding/Coding-Android/src/master/app/src/main/res/drawable-hdpi for 127.0.0.1
2016/01/12 21:51:22 [D] Session ID: 9ac3fbc3279ccc40
2016/01/12 21:51:22 [D] CSRF Token: z18BDQBepkYMtgkTW0UsyavaP0k6MTQ1MjYwMzkwMjMyMzI1MTM5MQ==
2016/01/12 21:51:30 [D] Template: repo/home
[Macaron] Completed /coding/Coding-Android/src/master/app/src/main/res/drawable-hdpi 200 OK in 8.47576734s
[Macaron] Started GET /avatars/2 for 127.0.0.1
[Macaron] [Static] Serving /2
[Macaron] Completed /avatars/2 304 Not Modified in 292.691µs

@unknwon yes , i guess it possible that my max open file limit didn't apply to the context that runs Gogs (start Gogs as service), but how do i make gogs apply max open file limit and work well in service mode (start Gogs as service)?

[simonszu]

@winsx Yes, this was exactly the same problem i had. Starting gogs manually works fine, starting gogs via systemd raises the "too many open files" error.
So i found out that systemd has a LimitNOFILE directive, and ignores the normal ulimit setting. Setting this directive works for me. I will create a pull request, i think.