Segfaults with heavy load on TCP / IO / Tasks

[despeset]

I'm getting segfaults reliably when putting the HttpServer package under heavy load.

To recreate, save this script as server.jl:

using HttpServer

http = HttpHandler() do req::Request, res::Response
    Response("-")
end

http.events["listen"]  = ( port ) -> println("Listening on $port...")

server = Server( http )
run( server, 8000 )

And this one to spam with curl:

#!/usr/bin/env bash
oi=0
REPS=200
echo "" > curl.txt
while true; do
        printf "\n"
        let TOTAL=$oi*$REPS
        printf "$TOTAL | "
        for i in $( seq 1 $REPS); do
                curl http://localhost:8000/hello/world 2>> ./curl.txt &
                printf "."
        done
        sleep 0.8
        FAILURES=$(tail -50 ./curl.txt | grep -c "7")
        if [ $FAILURES -gt 30 ]; then
                printf "Too many connection failures ($FAILURES/50), pausing for 10 seconds."
                sleep 10
        fi
        let oi=$oi+1
done

Then run:

$> julia server.jl &
[1] <PID>
Listening on 8000...
$> ./spam-with-curl.sh

That should work, but may depend a lot on the box in question. If results vary, you can definitely cause a failure with a higher-throughput load tester like siege or ab.

[Keno]

There's currently a libuv bug that causes an assertion failure on high load. Will fix soon. Is that what you're seeing?

[despeset]

Not sure but that sounds likely.

[Keno]

Alright, I'll update libuv (I had been trying to avoid it because they recently completely changed their error API, but I have to do it eventually, might as well do it now)

[Keno]

Hopefully fixed by 8109a6f. Could you confirm?

[despeset]

hmm.. I'm getting an error building:

$> make clean
...
$> make
...
configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."
make[2]: *** [libuv/config.status] Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

on OSX 10.7.5

[Keno]

pull again?

[despeset]

libtool: Version mismatch error.  This is libtool 2.4.2 Debian-2.4.2-1.2ubuntu1, but the
libtool: definition of this LT_INIT comes from libtool 2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2 Debian-2.4.2-1.2ubuntu1
libtool: and run autoconf again.
make[4]: *** [libuv_la-fs-poll.lo] Error 63
make[3]: *** [all] Error 2
make[2]: *** [libuv/libuv.a] Error 2
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

[Keno]

Yeah, we're on it. Libuv does weird things to their autoconf. Hold on.

[Keno]

Alright, try again.

[despeset]

Awesome! Builds great on an Ubuntu server box, and seems to have solved the segfault issue there. OSX build is still broken:

  GEN      include/uv-dtrace.h
  CC       libuv_la-fs-poll.lo
  CC       libuv_la-inet.lo
  CC       libuv_la-uv-common.lo
  CC       libuv_la-version.lo
  CC       libuv_la-async.lo
  CC       libuv_la-core.lo
  CC       libuv_la-dl.lo
  CC       libuv_la-fs.lo
src/unix/fs.c:210:34: warning: incompatible pointer types passing 'int
      (const struct dirent *)' to parameter of type 'int (*)(struct dirent *)'
      [-Wincompatible-pointer-types]
  n = scandir(req->path, &dents, uv__fs_readdir_filter, alphasort);
                                 ^~~~~~~~~~~~~~~~~~~~~
/usr/include/dirent.h:129:11: note: passing argument to parameter here
    int (*)(struct dirent *), int (*)(const void *, const void *)) ...
          ^
1 warning generated.
  CC       libuv_la-getaddrinfo.lo
  CC       libuv_la-loop-watcher.lo
  CC       libuv_la-loop.lo
  CC       libuv_la-pipe.lo
  CC       libuv_la-poll.lo
  CC       libuv_la-process.lo
  CC       libuv_la-signal.lo
  CC       libuv_la-stream.lo
  CC       libuv_la-tcp.lo
  CC       libuv_la-thread.lo
  CC       libuv_la-threadpool.lo
  CC       libuv_la-timer.lo
  CC       libuv_la-tty.lo
  CC       libuv_la-udp.lo
  CC       libuv_la-darwin.lo
  CC       libuv_la-darwin-proctitle.lo
  CC       libuv_la-fsevents.lo
  CC       libuv_la-kqueue.lo
  CC       libuv_la-proctitle.lo
  CCLD     libuv.la
 ./install-sh -c -d '/usr/local/src/julia/usr/lib'
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libuv.la '/usr/local/src/julia/usr/lib'
libtool: install: /usr/bin/install -c .libs/libuv.11.dylib /usr/local/src/julia/usr/lib/libuv.11.dylib
libtool: install: (cd /usr/local/src/julia/usr/lib && { ln -s -f libuv.11.dylib libuv.dylib || { rm -f libuv.dylib && ln -s libuv.11.dylib libuv.dylib; }; })
libtool: install: /usr/bin/install -c .libs/libuv.lai /usr/local/src/julia/usr/lib/libuv.la
libtool: install: /usr/bin/install -c .libs/libuv.a /usr/local/src/julia/usr/lib/libuv.a
libtool: install: chmod 644 /usr/local/src/julia/usr/lib/libuv.a
libtool: install: ranlib /usr/local/src/julia/usr/lib/libuv.a
 ./install-sh -c -d '/usr/local/src/julia/usr/include'
 /usr/bin/install -c -m 644 include/uv.h include/uv-errno.h include/uv-unix.h include/uv-darwin.h '/usr/local/src/julia/usr/include'
install_name_tool: input file: /usr/local/src/julia/usr/lib/libuv.a is not a Mach-O file
make[2]: *** [/usr/local/src/julia/usr/lib/libuv.a] Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

[StefanKarpinski]

Most people on OS X seem to be getting this. There's a thread about it on the dev list where I posted a link to a pre-breakage commit as well.

[Keno]

Sorry about that my OS X is broken at the moment. @vtjnash Could you see what's going on?

[StefanKarpinski]

#3854

[despeset]

The fix for #3854 solved the build issue on OS X – but unfortunately the segfaults persist. I just recreated it with the exact instructions at the top of this issue. Bummer.

How can I provide better diagnostic information on this beyond "It's a segmentation fault"?

[StefanKarpinski]

You can get more info from make debug and then run Julia under gdb. I can give more details if necessary.

[despeset]

Fresh build as of 2eb48e5 seems stable.