Major issue: "Child Died!" Stud Dies

[mitchellh]

I ran both 0.2 tag as well as 4a2311d in production with the following parameters:

stud --ssl -n 2 -b 127.0.0.1,8443 -f *,443 -u stud --write-proxy /etc/ssl/cert.pem

And about once every 10 minutes I would receive this in the log:

{backend} Connection closed
{backend} Connection closed
{backend} Connection closed
{backend} Connection closed
{backend} Connection closed
{client} Unexpected SSL error (in handshake): 5
{backend} Connection closed
{backend} Connection closed
{backend} Connection closed
{core} A child (25298) died!  This should not happen! Goodbye cruel world!

Once that child dies then both children appear to die and stud stops accepting connections. Am I configuring something incorrectly? This appears to be a pretty major bug otherwise.

Thanks!

[mitchellh]

To make things simpler, I dropped down to use only 1 process (not specifying a -n) and I'm still seeing the child die at least once every minute. This is my syslog output:

Oct  4 20:58:34 production-east-lb-001 init: lb-stud main process (27486) killed by TERM signal
Oct  4 20:58:34 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:58:47 production-east-lb-001 init: lb-stud main process (27489) killed by TERM signal
Oct  4 20:58:47 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:16 production-east-lb-001 init: lb-stud main process (27497) killed by TERM signal
Oct  4 20:59:16 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:29 production-east-lb-001 init: lb-stud main process (27518) killed by TERM signal
Oct  4 20:59:29 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:36 production-east-lb-001 init: lb-stud main process (27520) killed by TERM signal
Oct  4 20:59:36 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:44 production-east-lb-001 init: lb-stud main process (27522) killed by TERM signal
Oct  4 20:59:44 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:50 production-east-lb-001 init: lb-stud main process (27524) killed by TERM signal
Oct  4 20:59:50 production-east-lb-001 init: lb-stud main process ended, respawning
Oct  4 20:59:58 production-east-lb-001 init: lb-stud main process (27531) killed by TERM signal
Oct  4 20:59:58 production-east-lb-001 init: lb-stud main process ended, respawning

I otherwise am running with the same options given above.

[mitchellh]

This doesn't happen every time but I am intermittently seeing this as well:

stud: stud.c:690: handle_accept: Assertion `(*__errno_location ()) == 4 || (*__errno_location ()) == 11 || (*__errno_location ()) == 11' failed.

[BMatthieu]

Hi mitchellh,

I've the same error with stud:

stud: stud.c:690: handle_accept: Assertion `(___errno_location ()) == 4 || (___errno_location ()) == 11 || (*__errno_location ()) == 11' failed.

But for me stud is stopped every days, one crash by day.

I haven't no more log than you. We have 4 stud servers and two have the problem but the others run perfectly since 2 weeks.

[vincentbernat]

Hi!

A core file would help (ulimit -c unlimited before running stud).

[Neopallium]

I ran into that assertion when doing some stress testing. For me it was cause by stud running out of file descriptors (accept() was returning errno == EMFILE "Too many open files"). Check ulimit -n to see what your per-process limit is.

[BMatthieu]

Hi,

Thanks a lot. In fact the solution is to set ulimit.

Matthieu

[mitchellh]

Yep! I opened a new issue #41 for this specific issue.

[jamwt]

Okay, fixed #41.