Erlang and file descriptors (or: why I only got blank pages) #295

Closed
kaos opened this Issue Feb 20, 2012 · 7 comments

Comments

Projects
None yet
2 participants
Owner

kaos commented Feb 20, 2012

The resolution to this issue lies in documentation or wiki or the like, I guess.

I post it here for now to bring it to attention.
If the core dev team already was very well aware of this, I feel it's time it was communicated to the rest of the zotonic user base.

I noticed I got a lot of completely empty responses from a zotonic instance (regardless of which site I was accessing).
Looking at the logs, I found pages upon pages of errors like this:

=ERROR REPORT==== 20-Feb-2012::20:59:26 ===
{mochiweb_socket_server,317,{acceptor_error,{error,accept_failed}}}

=ERROR REPORT==== 20-Feb-2012::20:59:26 ===
    application: mochiweb
    "Accept failed error"
    "{error,emfile}"

=ERROR REPORT==== 20-Feb-2012::20:59:27 ===
{mochiweb_socket_server,317,{acceptor_error,{error,accept_failed}}}

=INFO REPORT==== 20-Feb-2012::20:59:34 ===
Could not find template: article.tpl (enoent)
=ERROR REPORT==== 20-Feb-2012::20:59:34 ===
webmachine error: path="/en/article/684/chuck-moore-on-the-lost-art-of-keeping-it-simple"
{error,{throw,{error,{template_not_found,"article.tpl",enoent}},
              [{z_template,render,3},
               {resource_page,html,1},
               {resource_page,to_html,2},
               {webmachine_resource,resource_call,3},
               {webmachine_resource,do,3},
               {webmachine_decision_core,decision,3},
               {webmachine_decision_core,handle_request,2},
               {webmachine_mochiweb,loop,2}]}}

I know that article.tpl exists, so a little google to assist, turning up this nice blog post: http://metajack.im/2008/09/23/file-descriptors-are-yummy-or-common-pitfalls-of-ejabberd/

Looking at my ulimit, it says it is set to 1024, and checking how much beam has eaten up gives me slightly more...?!

[root@ganesha sasl]# lsof | grep beam | wc -l
1093

which is a considerable amount out of the total 2457 lines that lsof spits out.
I'm wondering whether there can be a fd leak...? Better keep an eye on it.

A quick reset of all beam processes trunc's it down to 114:

[root@ganesha sasl]# kill `pidof beam`
[root@ganesha sasl]# lsof | grep beam | wc -l
114

And my pages serve just fine again.

Owner

mmzeeman commented Feb 20, 2012

Was your site busy? Or was there another problem.

And yes, you are totally right, this derives a cookbook entry.

There is indeed the file descriptors. How you up them depends on your os, heck no, that even differs between linux distro's. For ubuntu it goes like this:

Edit /etc/security/limits.conf and add an entry like this for the zotonic user.

zotonic soft nofile 10000
zotonic hard nofile 20000

Another thing you have to deal with is the maximum open ports beam can handle for you. Every sockets is an open port, so just increasing ulimit won't help. The default value for that is also 1024. There is a setting for that in: src/scripts/helpers/zotonic_setup

Owner

kaos commented Feb 20, 2012

Not very busy. But actually serving more requests than I expected.

[root@ganesha ~]# cat /opt/zotonic/zotonic-dev/priv/log/access.log.2012_02_20_19 | wc -l
363

Thanks for the hint to the config option in zotonic_setup.

Owner

mmzeeman commented Feb 20, 2012

Huh? That is not much indeed. I've seen at least a million of requests in an hour with the 1024 setting.

Owner

mmzeeman commented Feb 21, 2012

Are you sure it is the incoming netwerk connections? Or can there be something else eating fds?

You can count the open network connections on your system with this command. netstat ap | grep -c EST

Owner

kaos commented Feb 21, 2012

It could be anything.
And as you suspect, it may very well be something else than incoming TCP connections.
What I did see in abundance in the output from lsof | grep beam was pipe's.

So, perhaps a misbehaving port driver...

Owner

mmzeeman commented Feb 21, 2012

Could still be the sockets. They are handled by port drivers too.

Owner

kaos commented Dec 4, 2012

No action required for this specific report at this time.

@kaos kaos closed this Dec 4, 2012

@ddeboer ddeboer referenced this issue May 3, 2017

Merged

doc: Add server configuration chapter #1680

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment