Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Unit tests crash emulator on fresh checkout #11

Closed
kevsmith opened this Issue · 5 comments

2 participants

@kevsmith

First off, thanks for writing erlzmq2. I have big plans for this app :)

When I do a fresh checkout and build via 'make && make test' I get the following crash:

==> erlzmq2 (compile)
==> erlzmq2 (eunit)
Assertion failed: nbytes == sizeof (command_t) (mailbox.cpp:194)
Assertion failed: false (object.cpp:128)
make: *** [test] Abort trap

This is on a late-2009 MBP w/latest Snow Leopard. My Erlang environment is a 64 bit SMP-enabled build of R14B03. Yurii had asked me to look at my ulimit and set it to 1200. I checked and my default ulimit is 10k:

perdido:erlzmq2 ksmith$ ulimit -n
10000

I'm getting the same failure on my fork with the rebar build fixes, too, so it could be environmental or something not getting detected correctly as part of the build.

@kevsmith

Dug into this a bit further and determined the code is tripping over an assert in ZeroMQ itself in zmq::mailbox_t::send on line 194:

//  This should never happen as we've already checked that command size is
//  less than PIPE_BUF.
zmq_assert (nbytes == sizeof (command_t));

I'm not at all familiar with ZeroMQ internals but it looks like some sort of fragmented send.

@kevsmith

I managed to narrow the crash down to shutdown_stress_test. If I configure the test to create < 46 processes it succeeds the majority of the time with just a few intermittent crashes. Dropping the processes below 30 allows it to pass every time. Anything above 46 processes crashes every single time.

I sprinkled some printf() calls in zmq::mailbox_t::send() and have been able to confirm the crash is due to a short send:

nbytes: 48, sizeof(command_t): 48
Invalid argument
rc == 0 (mailbox.cpp:179)
nbytes: 32, sizeof(command_t): 48
Assertion failed: nbytes == sizeof (command_t) (mailbox.cpp:196)
make: *** [test] Abort trap

I suspect some sort of race where the rep socket is getting closed before the req socket can xmit all its data. Will continue to poke at this as time allows.

@kevsmith

Looks like the short sends are being triggered when a socket is either closed or termed. The NIFs and background thread aren't synchronized when either of these ops occur so I think the zmq sockets are getting torn down while still in use, thus causing the short sends. I added a ErlNifCond to erlzmq_socket_t and used it to coordinate closing between erlzmq_nif_close and the worker thread. This seemed to improve stability but didn't eliminate the crashes.

@evax
Owner

Hi Kevin,
Sorry for the late answer, I think the solution is to be found on the zeromq tuning page.
Apart from the ulimit change, here's what's stated:

0mq uses socketpairs for internal signaling and communications.
To avoid overrunning this resource, expand these buffers.
Edit the /etc/sysctl.conf file and add two lines and reboot:
    net.local.stream.sendspace=3000000
    net.local.stream.recvspace=3000000
@kevsmith

Thanks for the pointer to the relevant sysctl entries. Making the recommended edits seems to have fixed the problem completely. I can now run back-to-back sets of unit tests without a single crash.

@kevsmith kevsmith closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.