Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Erlang and file descriptors (or: why I only got blank pages) #295

Closed
kaos opened this Issue · 7 comments

2 participants

@kaos
Owner

The resolution to this issue lies in documentation or wiki or the like, I guess.

I post it here for now to bring it to attention.
If the core dev team already was very well aware of this, I feel it's time it was communicated to the rest of the zotonic user base.

I noticed I got a lot of completely empty responses from a zotonic instance (regardless of which site I was accessing).
Looking at the logs, I found pages upon pages of errors like this:

=ERROR REPORT==== 20-Feb-2012::20:59:26 ===
{mochiweb_socket_server,317,{acceptor_error,{error,accept_failed}}}

=ERROR REPORT==== 20-Feb-2012::20:59:26 ===
    application: mochiweb
    "Accept failed error"
    "{error,emfile}"

=ERROR REPORT==== 20-Feb-2012::20:59:27 ===
{mochiweb_socket_server,317,{acceptor_error,{error,accept_failed}}}

=INFO REPORT==== 20-Feb-2012::20:59:34 ===
Could not find template: article.tpl (enoent)
=ERROR REPORT==== 20-Feb-2012::20:59:34 ===
webmachine error: path="/en/article/684/chuck-moore-on-the-lost-art-of-keeping-it-simple"
{error,{throw,{error,{template_not_found,"article.tpl",enoent}},
              [{z_template,render,3},
               {resource_page,html,1},
               {resource_page,to_html,2},
               {webmachine_resource,resource_call,3},
               {webmachine_resource,do,3},
               {webmachine_decision_core,decision,3},
               {webmachine_decision_core,handle_request,2},
               {webmachine_mochiweb,loop,2}]}}

I know that article.tpl exists, so a little google to assist, turning up this nice blog post: http://metajack.im/2008/09/23/file-descriptors-are-yummy-or-common-pitfalls-of-ejabberd/

Looking at my ulimit, it says it is set to 1024, and checking how much beam has eaten up gives me slightly more...?!

[root@ganesha sasl]# lsof | grep beam | wc -l
1093

which is a considerable amount out of the total 2457 lines that lsof spits out.
I'm wondering whether there can be a fd leak...? Better keep an eye on it.

A quick reset of all beam processes trunc's it down to 114:

[root@ganesha sasl]# kill `pidof beam`
[root@ganesha sasl]# lsof | grep beam | wc -l
114

And my pages serve just fine again.

@mmzeeman
Owner

Was your site busy? Or was there another problem.

And yes, you are totally right, this derives a cookbook entry.

There is indeed the file descriptors. How you up them depends on your os, heck no, that even differs between linux distro's. For ubuntu it goes like this:

Edit /etc/security/limits.conf and add an entry like this for the zotonic user.

zotonic soft nofile 10000
zotonic hard nofile 20000

Another thing you have to deal with is the maximum open ports beam can handle for you. Every sockets is an open port, so just increasing ulimit won't help. The default value for that is also 1024. There is a setting for that in: src/scripts/helpers/zotonic_setup

@kaos
Owner

Not very busy. But actually serving more requests than I expected.

[root@ganesha ~]# cat /opt/zotonic/zotonic-dev/priv/log/access.log.2012_02_20_19 | wc -l
363

Thanks for the hint to the config option in zotonic_setup.

@mmzeeman
Owner

Huh? That is not much indeed. I've seen at least a million of requests in an hour with the 1024 setting.

@mmzeeman
Owner

Are you sure it is the incoming netwerk connections? Or can there be something else eating fds?

You can count the open network connections on your system with this command. netstat ap | grep -c EST

@kaos
Owner

It could be anything.
And as you suspect, it may very well be something else than incoming TCP connections.
What I did see in abundance in the output from lsof | grep beam was pipe's.

So, perhaps a misbehaving port driver...

@mmzeeman
Owner

Could still be the sockets. They are handled by port drivers too.

@kaos
Owner

No action required for this specific report at this time.

@kaos kaos closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.