Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hang building documentation on racketcs #2725

Closed
samth opened this issue Jul 4, 2019 · 15 comments
Closed

Hang building documentation on racketcs #2725

samth opened this issue Jul 4, 2019 · 15 comments

Comments

@samth
Copy link
Sponsor Member

samth commented Jul 4, 2019

I consistently get a hang with parallel builds of racketcs on linux. The hang happens here:

raco setup: 0 running: <pkgs>/racket-index/scribblings/main/local-redirect.scrbl
raco setup: 4 running: <pkgs>/racket-index/scribblings/main/search.scrbl

and the stack when hanging looks like:

Thread 1 "racketcs" received signal SIGINT, Interrupt.
0x00007fdf8f596729 in __GI___poll (fds=0x561d82185030, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29      in ../sysdeps/unix/sysv/linux/poll.c

The stack as printed by gdb is pretty useless, but I can disassemble the relevant functions if needed.

@mflatt
Copy link
Member

mflatt commented Jul 5, 2019

Unfortunately, I haven't been able to replicate the problem, so far.

It makes sense that the main thread would be blocked in poll. Are other OS threads for other places still running? If so, can you get stack traces for other threads? (The relevant GDB commands are something like thread list and thread select.)

@samth
Copy link
Sponsor Member Author

samth commented Jul 5, 2019

For reference, I had to use info threads to list all the threads. Here they are:

(gdb) info threads
  Id   Target Id                                   Frame
* 1    Thread 0x7f832eb4a780 (LWP 5892) "racketcs" 0x00007f832ec60729 in __GI___poll (fds=0x55e601b696e0, nfds=2,
    timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  2    Thread 0x7f832d8da700 (LWP 6424) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645cc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f832d0d9700 (LWP 6425) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  4    Thread 0x7f831f7fe700 (LWP 6426) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  5    Thread 0x7f832c8d8700 (LWP 6427) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  6    Thread 0x7f831effd700 (LWP 6428) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0x7f82ee1cb700 (LWP 6429) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645cc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  8    Thread 0x7f82e62cf700 (LWP 6430) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645cc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7f82daf6b700 (LWP 6431) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x55e601b645cc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  29   Thread 0x7f830ffff700 (LWP 6565) "racketcs" 0x00007f832ed4d440 in __GI___nanosleep (
    requested_time=0x7f830fffeba0, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28

Waiting a little while and then breaking again gives similar results.

Here's the stack for thread 2:

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f832d8da700 (LWP 6424))]
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55e601b645cc)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88      ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) where
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55e601b645cc)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55e601b61bc0, cond=0x55e601b645a0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55e601b645a0, mutex=0x55e601b61bc0) at pthread_cond_wait.c:655
#3  0x000055e60022c69b in ?? ()
#4  0x0000000040cce46c in ?? ()
#5  0x00007f832800df00 in ?? ()
#6  0x000055e60022b3c8 in ?? ()
#7  0x000055e60022b45c in ?? ()
#8  0x000055e60022c33b in ?? ()
#9  0x00007f832ed43182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f832ec6cb1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The other similar stacks are very similar, with the same stack frame address in all of them except frame 5.

The last thread is different, here's the stack trace:

(gdb) thread 29
[Switching to thread 29 (Thread 0x7f830ffff700 (LWP 6565))]
#0  0x00007f832ed4d440 in __GI___nanosleep (requested_time=0x7f830fffeba0, remaining=0x0)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28      ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
(gdb) where
#0  0x00007f832ed4d440 in __GI___nanosleep (requested_time=0x7f830fffeba0, remaining=0x0)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x000055e600254d30 in ?? ()
#2  0x0000000040d0ceb9 in ?? ()
#3  0x00007f81e0078470 in ?? ()
#4  0x000055e60022b3c8 in ?? ()
#5  0x00007f825bfd6f04 in ?? ()
#6  0x00007f82c3a65add in ?? ()
#7  0x000055e601b5f8c0 in ?? ()
#8  0x00007f82c3885ec0 in ?? ()
#9  0x00000000409037db in ?? ()
#10 0x0000000000001000 in ?? ()
#11 0x00007f81e001d090 in ?? ()
#12 0x00007f81e001d090 in ?? ()
#13 0x0000000000000000 in ?? ()

@mflatt
Copy link
Member

mflatt commented Jul 5, 2019

My only thought at the moment is that it might be an interaction of futures and places. What happens if you comment out the (host:sleep 0.001) at "racket/src/thread/future.rkt" line 596? I'm specifically wondering whether the nanosleep call persists in a thread's stack trace.

@samth
Copy link
Sponsor Member Author

samth commented Jul 9, 2019

I commented out that call, got another hang (at a different place in the doc build), and the nanosleep thread went away replaced by a thread with this "helpful" stack trace:

(gdb) info threads
  Id   Target Id                                   Frame
* 1    Thread 0x7f7560fb3780 (LWP 2583) "racketcs" 0x00007f75610c9729 in __GI___poll (fds=0x556e8b924280, nfds=2,
    timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  2    Thread 0x7f755fd41700 (LWP 2687) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f755f540700 (LWP 2688) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  4    Thread 0x7f755ed3f700 (LWP 2689) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  5    Thread 0x7f755d53c700 (LWP 2690) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  6    Thread 0x7f755e538700 (LWP 2691) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0x7f753d3a7700 (LWP 2692) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  8    Thread 0x7f753cba6700 (LWP 2693) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7f753b56b700 (LWP 2694) "racketcs" futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x556e8b8965fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  10   Thread 0x7f75469d9700 (LWP 2709) "racketcs" 0x0000000042e321dd in ?? ()
(gdb) thread 10
[Switching to thread 10 (Thread 0x7f75469d9700 (LWP 2709))]
#0  0x0000000042e321dd in ?? ()
(gdb) where
#0  0x0000000042e321dd in ?? ()
#1  0x00007f7490022a60 in ?? ()
#2  0x0000556e898063c8 in ?? ()
#3  0x0000556e8980645c in ?? ()
#4  0x0000556e8980733b in ?? ()
#5  0x00007f75611ac182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f75610d5b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@mflatt
Copy link
Member

mflatt commented Jul 11, 2019

That is helpful, and I'm pretty sure we're on the right track.

I've pushed a commit that fixes one synchronization problem. I'm not optimistic that I've found the right problem, though. More generally, I'm skeptical that the ad hoc, CAS-based synchronization for custodians and futures is a good idea, and probably there's a better approach.

@pmatos
Copy link
Collaborator

pmatos commented Jul 13, 2019

I started to recently see this on a regular basis. Let me know if I can provide any further information besides what @samth already provided.

@mflatt
Copy link
Member

mflatt commented Jul 18, 2019

I was finally able to provoke similar behavior on my machine, and the latest commit fixes several problems. Please let me know whether you see any change.

@mflatt
Copy link
Member

mflatt commented Aug 16, 2019

@samth and @pmatos: Just checking in... Have you seen this happen again recently?

@pmatos
Copy link
Collaborator

pmatos commented Aug 16, 2019

@mflatt Nope, not recently. I built a 7.4cs from scratch today and didn't face the issue.

@samth
Copy link
Sponsor Member Author

samth commented Aug 18, 2019

I haven't seen this recently either.

@samth samth closed this as completed Aug 18, 2019
@samth
Copy link
Sponsor Member Author

samth commented Oct 8, 2019

This has started happening again. Here's the current best stack information I have:

Thread 1 "racketcs" received signal SIGINT, Interrupt.
[Switching to Thread 0x7f40a7c8f780 (LWP 10671)]
0x00007f40a7da5729 in __GI___poll (fds=0x55695d08ec80, nfds=2, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
29	in ../sysdeps/unix/sysv/linux/poll.c
(gdb) where
#0  0x00007f40a7da5729 in __GI___poll (fds=0x55695d08ec80, nfds=2, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
#3  0x000055695cfa7b90 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x000055695c7db680 in ?? ()
#6  0x000055695c7d055f in ?? ()
#7  0x000055695c7ceeea in ?? ()
#8  0x00007f40a7cbab6b in __libc_start_main (main=0x55695c7cecf0, argc=12, argv=0x7ffedae3b888, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffedae3b878)
    at ../csu/libc-start.c:308
#9  0x000055695c7cf06a in ?? ()
(gdb) 

Below is all the threads.

Thread 40 (Thread 0x7f404d1b3700 (LWP 12463)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f400c0c0560 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 39 (Thread 0x7f404fc07700 (LWP 12462)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f20000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 38 (Thread 0x7f4059f4b700 (LWP 12461)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f1c000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 37 (Thread 0x7f405d7bb700 (LWP 12460)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f28000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 36 (Thread 0x7f406a14b700 (LWP 12459)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/uni--Type <RET> for more, q to quit, c to continue without paging--
x/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f34000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 35 (Thread 0x7f407428b700 (LWP 12458)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f24000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 34 (Thread 0x7f4085093700 (LWP 12457)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f2c000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 33 (Thread 0x7f408b5f3700 (LWP 12456)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f3fa81538e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f3fa808c420, cond=0x7f3fa81538c0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f3fa81538c0, mutex=0x7f3fa808c420) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f30000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 31 (Thread 0x7f408346b700 (LWP 11472)):
#0  0x00007f40a7da5729 in __GI___poll (fds=0x7f3f1807d7b0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x00007f3f180aa9e0 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x00007f3f6e1c9174 in ?? ()
#6  0x00007f3f6da7dcfd in ?? ()
#7  0x000055695d026e40 in ?? ()
#8  0x00007f3f6e6076d8 in ?? ()
#9  0x0000000041e0783b in ?? ()
#10 0x0000000000001000 in ?? ()
#11 0x00007f3f180eb700 in ?? ()
#12 0x00007f3f180eb700 in ?? ()
#13 0x00007f3f6e607660 in ?? ()
#14 0x000055695d026e40 in ?? ()
#15 0x000055695c7d1d4e in ?? ()
#16 0x000055695c7d810d in ?? ()
#17 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#18 0x00007f3f6e1c89e5 in ?? ()
#19 0x00007f3f181177d0 in ?? ()
#20 0x000055695c7db488 in ?? ()
#21 0x00007f3f6e54e8d4 in ?? ()
#22 0x00007f3f6e35461d in ?? ()
#23 0x000055695d026e40 in ?? ()
#24 0x00007f3f6dfda950 in ?? ()
#25 0x0000000041e0783b in ?? ()
#26 0x0000000000001000 in ?? ()
#27 0x00007f3f18da61b0 in ?? ()
#28 0x00007f3f18da61b0 in ?? ()
#29 0x000055695c7d274c in ?? ()
#30 0x00007f3f6dfdd71d in ?? ()
#31 0x00007f3f6dfda750 in ?? ()
#32 0x000055695c7d810d in ?? ()
#33 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#34 0x00007f3f6e54e305 in ?? ()
#35 0x00007f3f190a9ae0 in ?? ()
#36 0x000055695c7db488 in ?? ()
#37 0x00007f3ff7750c34 in ?? ()
#38 0x00007f3ffd8010ad in ?? ()
#39 0x000055695d026e40 in ?? ()
#40 0x00007f3ffd7ab6d8 in ?? ()
#41 0x0000000041e0783b in ?? ()
#42 0x0000000000001000 in ?? ()
#43 0x00007f3f182e8dd0 in ?? ()
#44 0x00007f3f182e8dd0 in ?? ()
#45 0x000055695c7d274c in ?? ()
#46 0x00007f3ffd80b3bd in ?? ()
#47 0x00007f3ffd7ab4d8 in ?? ()
#48 0x000055695c7d810d in ?? ()
#49 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#50 0x00007f3ff7750665 in ?? ()
#51 0x00007f3f190a9bd0 in ?? ()
#52 0x000055695c7db488 in ?? ()
#53 0x00007f4016f47054 in ?? ()
#54 0x00007f3f6e31861d in ?? ()
#55 0x000055695d026e40 in ?? ()
#56 0x00007f3f772c5be0 in ?? ()
#57 0x0000000041e0783b in ?? ()
#58 0x0000000000001000 in ?? ()
#59 0x00007f3f182e8dd0 in ?? ()
#60 0x00007f3f182e8dd0 in ?? ()
#61 0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7f3ff1056700 (LWP 11471)):
#0  0x00007f40a7da5729 in __GI___poll (fds=0x7f3fdc1bbfc0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
#3  0x00007f3fdc0b2c60 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x00007f3f6e220c34 in ?? ()
#6  0x00007f3f6da5910d in ?? ()
#7  0x000055695d029940 in ?? ()
#8  0x00007f3f6ef351d0 in ?? ()
#9  0x0000000041e0783b in ?? ()
#10 0x0000000000001000 in ?? ()
#11 0x00007f3fdc084ec0 in ?? ()
#12 0x00007f3fdc084ec0 in ?? ()
#13 0x000055695c7d274c in ?? ()
#14 0x00007f3f6ef17b7d in ?? ()
#15 0x00007f3f6ef34fd0 in ?? ()
#16 0x000055695c7d810d in ?? ()
#17 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#18 0x00007f3f6e220665 in ?? ()
#19 0x00007f3fdc1aadf0 in ?? ()
#20 0x000055695c7db488 in ?? ()
#21 0x00007f3f6f7dc4c4 in ?? ()
#22 0x00007f3f6ea165bd in ?? ()
#23 0x000055695d029940 in ?? ()
#24 0x00007f3f6e940078 in ?? ()
#25 0x0000000041e0783b in ?? ()
#26 0x0000000000001000 in ?? ()
#27 0x00007f3fdc0d9c30 in ?? ()
#28 0x00007f3fdc0d9c30 in ?? ()
#29 0x000055695c7d274c in ?? ()
#30 0x00007f3f6e9dd6bd in ?? ()
#31 0x00007f3f6e97d478 in ?? ()
#32 0x000055695c7d810d in ?? ()
#33 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#34 0x00007f3f6f82eb45 in ?? ()
#35 0x00007f3fdc064e00 in ?? ()
#36 0x000055695c7db488 in ?? ()
#37 0x00007f3f6ec9cc34 in ?? ()
#38 0x00007f3f6dc77b6d in ?? ()
#39 0x000055695d029940 in ?? ()
#40 0x00007f3f6ee7f6d8 in ?? ()
#41 0x0000000041e0783b in ?? ()
#42 0x0000000000001000 in ?? ()
#43 0x00007f3fdc1c6ba0 in ?? ()
#44 0x00007f3fdc1c6ba0 in ?? ()
#45 0x000055695c7d274c in ?? ()
#46 0x00007f3f6de8073d in ?? ()
#47 0x00007f3f6ee7f4d8 in ?? ()
#48 0x000055695c7d810d in ?? ()
#49 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#50 0x00007f3f6ec9c665 in ?? ()
#51 0x00007f3fdc00c5e0 in ?? ()
#52 0x000055695c7db488 in ?? ()
#53 0x00007f3f6e54a414 in ?? ()
#54 0x00007f3f6e1a9b2d in ?? ()
#55 0x000055695d029940 in ?? ()
#56 0x00007f3f6e1c9be0 in ?? ()
#57 0x0000000041e0783b in ?? ()
#58 0x0000000000001000 in ?? ()
#59 0x00007f3fdc0b6540 in ?? ()
#60 0x00007f3fdc0b6540 in ?? ()
#61 0x000055695c7d274c in ?? ()
#62 0x00007f3f6e1d8bbd in ?? ()
#63 0x00007f3f6e1c99e0 in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#64 0x000055695c7d810d in ?? ()
#65 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#66 0x00007f3f6e549e45 in ?? ()
#67 0x00007f3fdc0dac40 in ?? ()
#68 0x000055695c7db488 in ?? ()
#69 0x00007f3f6e1c4fc4 in ?? ()
#70 0x00007f3f6dc3e88d in ?? ()
#71 0x000055695d029940 in ?? ()
#72 0x00007f3f6da2ab10 in ?? ()
#73 0x0000000041e0783b in ?? ()
#74 0x0000000000001000 in ?? ()
#75 0x00007f3fdc0dcb30 in ?? ()
#76 0x00007f3fdc0dcb30 in ?? ()
#77 0x000055695c7d274c in ?? ()
#78 0x00007f3f6da3f88d in ?? ()
#79 0x00007f3f6da2a910 in ?? ()
#80 0x000055695c7d810d in ?? ()
#81 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#82 0x00007f3f6e1c49f5 in ?? ()
#83 0x00007f3fdc1aa940 in ?? ()
#84 0x000055695c7db488 in ?? ()
#85 0x00007f3f6dd6ea04 in ?? ()
#86 0x00007f3f6dfddfdd in ?? ()
#87 0x000055695d029940 in ?? ()
#88 0x00007f3f6dc5ddd8 in ?? ()
#89 0x0000000041e0783b in ?? ()
#90 0x0000000000001000 in ?? ()
#91 0x00007f3fdc0dcb30 in ?? ()
#92 0x00007f3fdc0dcb30 in ?? ()
#93 0x000055695c7d274c in ?? ()
#94 0x00007f3f6e10c8fd in ?? ()
#95 0x00007f3f6dc5dbd8 in ?? ()
#96 0x000055695c7d810d in ?? ()
#97 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#98 0x00007f3f6dd6e035 in ?? ()
#99 0x00007f3fdc0d7950 in ?? ()
#100 0x000055695c7db488 in ?? ()
#101 0x00007f3f6edba414 in ?? ()
#102 0x00007f3f6e49d0dd in ?? ()
#103 0x000055695d029940 in ?? ()
#104 0x00007f3f6e408270 in ?? ()
#105 0x0000000041e0783b in ?? ()
#106 0x0000000000001000 in ?? ()
#107 0x00007f3fdc0a6130 in ?? ()
#108 0x00007f3fdc0a6130 in ?? ()
#109 0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7f4086bf5700 (LWP 11470)):
#0  0x00007f40a7da5729 in __GI___poll (fds=0x7f3fe8037590, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
#3  0x00007f3fe807deb0 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x00007f3f6f311c24 in ?? ()
#6  0x00007f3f6e033afd in ?? ()
#7  0x000055695d029680 in ?? ()
#8  0x00007f3f6de368b8 in ?? ()
#9  0x0000000041e0783b in ?? ()
#10 0x0000000000001000 in ?? ()
#11 0x00007f3fe80275e0 in ?? ()
#12 0x00007f3fe80275e0 in ?? ()
#13 0x000055695c7d274c in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00007f3f6e0d828d in ?? ()
#15 0x00007f3f6de366b8 in ?? ()
#16 0x000055695c7d810d in ?? ()
#17 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#18 0x00007f3f6f311655 in ?? ()
#19 0x00007f3fe80298b0 in ?? ()
#20 0x000055695c7db488 in ?? ()
#21 0x00007f3f6e035f14 in ?? ()
#22 0x00007f3f6d950a7d in ?? ()
#23 0x000055695d029680 in ?? ()
#24 0x00007f3f6e7d5be0 in ?? ()
#25 0x0000000041e0783b in ?? ()
#26 0x0000000000001000 in ?? ()
#27 0x00007f3fe801b660 in ?? ()
#28 0x00007f3fe801b660 in ?? ()
#29 0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7f407dfc3700 (LWP 11469)):
#0  0x00007f40a7da5729 in __GI___poll (fds=0x7f3fa82f6400, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
#3  0x00007f3fa813db20 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x0000000079cc4c34 in ?? ()
#6  0x00007f3ff40c66ad in ?? ()
#7  0x000055695d0b1d70 in ?? ()
#8  0x00007f4065788f98 in ?? ()
#9  0x0000000041e0783b in ?? ()
#10 0x0000000000001000 in ?? ()
#11 0x00007f3fa83e0d30 in ?? ()
#12 0x00007f3fa83e0d30 in ?? ()
#13 0x000055695c7d274c in ?? ()
#14 0x00007f3ff41981fd in ?? ()
#15 0x00007f4065788d98 in ?? ()
#16 0x000055695c7d810d in ?? ()
#17 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#18 0x0000000079cc4665 in ?? ()
#19 0x00007f3fa82a4330 in ?? ()
#20 0x000055695c7db488 in ?? ()
#21 0x00000000745b1af4 in ?? ()
#22 0x00007f3f6d8905ed in ?? ()
#23 0x000055695d0b1d70 in ?? ()
#24 0x00007f3f88c3c270 in ?? ()
#25 0x0000000041e0783b in ?? ()
#26 0x0000000000001000 in ?? ()
#27 0x00007f3fa82487b0 in ?? ()
#28 0x00007f3fa82487b0 in ?? ()
#29 0x000055695c7d274c in ?? ()
#30 0x00007f3f88ae837d in ?? ()
#31 0x00007f3f88c3c070 in ?? ()
#32 0x000055695c7d810d in ?? ()
#33 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#34 0x00000000745b1525 in ?? ()
#35 0x00007f3fa82d1730 in ?? ()
#36 0x000055695c7db488 in ?? ()
#37 0x00007f3f76748fc4 in ?? ()
#38 0x00007f3f6e17129d in ?? ()
#39 0x000055695d0b1d70 in ?? ()
#40 0x00007f3f6dc84f98 in ?? ()
#41 0x0000000041e0783b in ?? ()
#42 0x0000000000001000 in ?? ()
#43 0x00007f3fa822f800 in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#44 0x00007f3fa822f800 in ?? ()
#45 0x000055695c7d274c in ?? ()
#46 0x00007f3f6dc363bd in ?? ()
#47 0x00007f3f6dc84d98 in ?? ()
#48 0x000055695c7d810d in ?? ()
#49 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#50 0x00007f3f767489f5 in ?? ()
#51 0x00007f3fa81ad940 in ?? ()
#52 0x000055695c7db488 in ?? ()
#53 0x00007f3f708718f4 in ?? ()
#54 0x00007f3f6e42154d in ?? ()
#55 0x000055695d0b1d70 in ?? ()
#56 0x00007f3f6e046bc8 in ?? ()
#57 0x0000000041e0783b in ?? ()
#58 0x0000000000001000 in ?? ()
#59 0x00007f3fa8284690 in ?? ()
#60 0x00007f3fa8284690 in ?? ()
#61 0x000055695c7d274c in ?? ()
#62 0x00007f3f6e3d14fd in ?? ()
#63 0x00007f3f6e046750 in ?? ()
#64 0x000055695c7d810d in ?? ()
#65 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#66 0x00007f3f70871325 in ?? ()
#67 0x00007f3fa80a8f30 in ?? ()
#68 0x000055695c7db488 in ?? ()
#69 0x00007f3f6f788c34 in ?? ()
#70 0x00007f3f6da95b5d in ?? ()
#71 0x000055695d0b1d70 in ?? ()
#72 0x00007f3f6eca36d8 in ?? ()
#73 0x0000000041e0783b in ?? ()
#74 0x0000000000001000 in ?? ()
#75 0x00007f3fa8284690 in ?? ()
#76 0x00007f3fa8284690 in ?? ()
#77 0x000055695c7d274c in ?? ()
#78 0x00007f3f6d9d677d in ?? ()
#79 0x00007f3f6eca34d8 in ?? ()
#80 0x000055695c7d810d in ?? ()
#81 0x00007f408c426b11 in jpeg_CreateDecompress () from /lib/x86_64-linux-gnu/libjpeg.so.8
#82 0x00007f3f6f788665 in ?? ()
#83 0x00007f3fa8130f40 in ?? ()
#84 0x000055695c7db488 in ?? ()
#85 0x00007f3f6f0fd724 in ?? ()
#86 0x00007f3f6e7545ed in ?? ()
#87 0x000055695d0b1d70 in ?? ()
#88 0x00007f3f6ea6a950 in ?? ()
#89 0x0000000041e0783b in ?? ()
#90 0x0000000000001000 in ?? ()
#91 0x00007f3fa813cb10 in ?? ()
#92 0x00007f3fa813cb10 in ?? ()
#93 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f40875fa700 (LWP 11381)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f38000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f4087dfb700 (LWP 11380)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3f68000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f4089b27700 (LWP 11379)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f3fb4000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f408ade3700 (LWP 11378)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f40a0031d60 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f40a4a19700 (LWP 11377)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f4094000b20 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

--Type <RET> for more, q to quit, c to continue without paging--
Thread 9 (Thread 0x7f40a621c700 (LWP 11376)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f4098015310 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f40a5a1b700 (LWP 11375)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f409c018090 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f40a521a700 (LWP 11374)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55695cfa77a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55695cf95bb0, cond=0x55695cfa7780) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55695cfa7780, mutex=0x55695cf95bb0) at pthread_cond_wait.c:655
#3  0x000055695c7dc76b in ?? ()
#4  0x0000000040dcecdc in ?? ()
#5  0x00007f4090010200 in ?? ()
#6  0x000055695c7db488 in ?? ()
#7  0x000055695c7db51c in ?? ()
#8  0x000055695c7dc40b in ?? ()
#9  0x00007f40a7e88182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f40a7db1b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f40a7c8f780 (LWP 10671)):
#0  0x00007f40a7da5729 in __GI___poll (fds=0x55695d08ec80, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055695c8319cc in ?? ()
#2  0x0000000042f8f022 in ?? ()
#3  0x000055695cfa7b90 in ?? ()
#4  0x000055695c7db488 in ?? ()
#5  0x000055695c7db680 in ?? ()
#6  0x000055695c7d055f in ?? ()
#7  0x000055695c7ceeea in ?? ()
#8  0x00007f40a7cbab6b in __libc_start_main (main=0x55695c7cecf0, argc=12, argv=0x7ffedae3b888, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffedae3b878) at ../csu/libc-start.c:308
#9  0x000055695c7cf06a in ?? ()

@samth samth reopened this Oct 8, 2019
@mflatt
Copy link
Member

mflatt commented Oct 8, 2019

One possibility is that the disable-interrupt and enable-interrupt recently added to with-global-lock in "lock.ss" is the wrong idea and creates a deadlock instead of avoiding it.

mflatt added a commit that referenced this issue Oct 9, 2019
Repair problems with asynchronous callbacks for futures and for
foreign callbacks. Asynchronous callbacks are used for future "sync"
operations, like `hash-set!`, that run must in a place's main thread
(as of commit f574583).

Separately, synchronization to clean up future threads used a `ping?`
flag in a backwards sense, and it also treated a record as a box.
These problems could cause place termination to hang.

Related to #2725
jbclements pushed a commit that referenced this issue Oct 10, 2019
Repair problems with asynchronous callbacks for futures and for
foreign callbacks. Asynchronous callbacks are used for future "sync"
operations, like `hash-set!`, that run must in a place's main thread
(as of commit f574583).

Separately, synchronization to clean up future threads used a `ping?`
flag in a backwards sense, and it also treated a record as a box.
These problems could cause place termination to hang.

Related to #2725

(cherry picked from commit 7cc3345)
@samth
Copy link
Sponsor Member Author

samth commented Oct 10, 2019

This hung again after the most recent changes, here's the backtraces of all threads.

(gdb) thread apply all where 2

Thread 22 (Thread 0x7fa9a769f700 (LWP 31781)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 21 (Thread 0x7fa9a8eb7700 (LWP 31780)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 20 (Thread 0x7fa9ab927700 (LWP 31779)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 19 (Thread 0x7fa9ad00f700 (LWP 31778)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 18 (Thread 0x7fa9af7a3700 (LWP 31777)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 17 (Thread 0x7fa9bb113700 (LWP 31776)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 16 (Thread 0x7faa23daf700 (LWP 31775)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 15 (Thread 0x7faa89085700 (LWP 31774)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa98c097708) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa98c0976b0, cond=0x7fa98c0976e0) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 14 (Thread 0x7faa7a4cb700 (LWP 31760)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x7fa99002c0f0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--c
(More stack frames follow...)

Thread 13 (Thread 0x7faa7d123700 (LWP 31759)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x7fa98c0bfc00, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
(More stack frames follow...)

Thread 12 (Thread 0x7faa7e3d3700 (LWP 31758)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x7fa99446bcf0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
(More stack frames follow...)

Thread 11 (Thread 0x7faa817af700 (LWP 31757)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x7fa9e8084700, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
(More stack frames follow...)

Thread 10 (Thread 0x7faa8b1e3700 (LWP 31755)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x7faa3c0b7ef0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
(More stack frames follow...)

Thread 9 (Thread 0x7faa6d32b700 (LWP 31753)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 8 (Thread 0x7faa82ddb700 (LWP 31752)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 7 (Thread 0x7faa9e7fc700 (LWP 31751)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 6 (Thread 0x7faa9effd700 (LWP 31750)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 5 (Thread 0x7faa9f7fe700 (LWP 31749)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 4 (Thread 0x7faa9ffff700 (LWP 31748)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 3 (Thread 0x7faaa56a9700 (LWP 31747)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 2 (Thread 0x7faaa4ea8700 (LWP 31746)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55ed121e3c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55ed121d43f0, cond=0x55ed121e3c20) at pthread_cond_wait.c:502
(More stack frames follow...)

Thread 1 (Thread 0x7faaa691b780 (LWP 30898)):
#0  0x00007faaa6a31729 in __GI___poll (fds=0x55ed1227e5e0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055ed118ed9cc in ?? ()
(More stack frames follow...)

@mflatt
Copy link
Member

mflatt commented Oct 11, 2019

I'm hopeful that this change will actually fix the problem: 61e3965

Broken caching could cause Racket CS to raise an error in atomic mode. If that happens during the doc-build phase, then the place where that error happens would become stuck, and so all of raco setup would become stuck at the end of the phase. Also, the bug in caching is related to locale conversions, which would be most relevant for non-Mac Unix path conversion. The bug also needs custodian shutdowns, like the ones that happen for sandboxes during documentation building. Finally, since the cache change is from a few weeks ago, that's consistent with a report from @samth that the problem started happening again a while back.

@pmatos
Copy link
Collaborator

pmatos commented Nov 12, 2019

I haven't seen this in awhile. Can we close?

@samth samth closed this as completed Jan 1, 2020
MarcKaufmann pushed a commit to MarcKaufmann/racket that referenced this issue Apr 15, 2020
MarcKaufmann pushed a commit to MarcKaufmann/racket that referenced this issue Apr 15, 2020
Repair problems with asynchronous callbacks for futures and for
foreign callbacks. Asynchronous callbacks are used for future "sync"
operations, like `hash-set!`, that run must in a place's main thread
(as of commit f574583).

Separately, synchronization to clean up future threads used a `ping?`
flag in a backwards sense, and it also treated a record as a box.
These problems could cause place termination to hang.

Related to racket#2725
@AlexKnauth AlexKnauth mentioned this issue Dec 13, 2021
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants