New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cherokee deadlocks #1208

Open
skinkie opened this Issue May 9, 2018 · 3 comments

Comments

Projects
None yet
1 participant
@skinkie
Member

skinkie commented May 9, 2018

On several systems I noticed that Cherokee sometimes hangs. A way to reproduce it has not been found yet.

(gdb) info threads
  Id   Target Id         Frame 
  40   Thread 0x7fc3bf429700 (LWP 12134) "cherokee-worker" 0x00007fc3d65d6eaf in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
  39   Thread 0x7fc3bfc2a700 (LWP 12133) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  38   Thread 0x7fc3c042b700 (LWP 12132) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  37   Thread 0x7fc3c0c2c700 (LWP 12131) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  36   Thread 0x7fc3c142d700 (LWP 12130) "cherokee-worker" 0x00007fc3d5a5e333 in epoll_wait () from /lib64/libc.so.6
  35   Thread 0x7fc3c1c2e700 (LWP 12129) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  34   Thread 0x7fc3c242f700 (LWP 12128) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  33   Thread 0x7fc3c2c30700 (LWP 12127) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  32   Thread 0x7fc3c3431700 (LWP 12126) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  31   Thread 0x7fc3c3c32700 (LWP 12125) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  30   Thread 0x7fc3c4433700 (LWP 12124) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
  29   Thread 0x7fc3c4c34700 (LWP 12123) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  28   Thread 0x7fc3c5435700 (LWP 12122) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  27   Thread 0x7fc3c5c36700 (LWP 12121) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  26   Thread 0x7fc3c6437700 (LWP 12120) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  25   Thread 0x7fc3c6c38700 (LWP 12119) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  24   Thread 0x7fc3c7439700 (LWP 12118) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  23   Thread 0x7fc3c7c3a700 (LWP 12117) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  22   Thread 0x7fc3c843b700 (LWP 12116) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  21   Thread 0x7fc3c8c3c700 (LWP 12115) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  20   Thread 0x7fc3c943d700 (LWP 12114) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  19   Thread 0x7fc3c9c3e700 (LWP 12113) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  18   Thread 0x7fc3ca43f700 (LWP 12112) "cherokee-worker" 0x00007fc3d65dc95c in---Type <return> to continue, or q <return> to quit---
 __lll_lock_wait () from /lib64/libpthread.so.0
  17   Thread 0x7fc3cac40700 (LWP 12111) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  16   Thread 0x7fc3cb441700 (LWP 12110) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  15   Thread 0x7fc3cbc42700 (LWP 12109) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  14   Thread 0x7fc3cc443700 (LWP 12108) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  13   Thread 0x7fc3ccc44700 (LWP 12107) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  12   Thread 0x7fc3cd445700 (LWP 12106) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  11   Thread 0x7fc3cdc46700 (LWP 12105) "cherokee-worker" 0x00007fc3d5a5e333 in epoll_wait () from /lib64/libc.so.6
  10   Thread 0x7fc3ce447700 (LWP 12104) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  9    Thread 0x7fc3cec48700 (LWP 12103) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  8    Thread 0x7fc3cf449700 (LWP 12102) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  7    Thread 0x7fc3cfc4a700 (LWP 12101) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
  6    Thread 0x7fc3d044b700 (LWP 12100) "cherokee-worker" 0x00007fc3d5a5e333 in epoll_wait () from /lib64/libc.so.6
  5    Thread 0x7fc3d0c4c700 (LWP 12099) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  4    Thread 0x7fc3d144d700 (LWP 12098) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  3    Thread 0x7fc3d1c4e700 (LWP 12097) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7fc3d244f700 (LWP 12096) "cherokee-worker" 0x00007fc3d65dc95c in __lll_lock_wait () from /lib64/libpthread.so.0
* 1    Thread 0x7fc3d6e71b40 (LWP 12092) "cherokee-worker" 0x00007fc3d65d743d in pthread_rwlock_wrlock () from /lib64/libpthread.so.0

(gdb) thread 1
[Switching to thread 1 (Thread 0x7fc3d6e71b40 (LWP 12092))]
#0  0x00007fc3d65d743d in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fc3d65d743d in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1  0x00007fc3d6a48abd in cherokee_bogotime_update () at bogotime.c:191
#2  0x00007fc3d67fff1d in cherokee_server_step (srv=0x6041c0) at server.c:1151
#3  0x00000000004016dc in main (argc=<optimized out>, argv=<optimized out>)
    at main_worker.c:407

(gdb) thread 40
[Switching to thread 40 (Thread 0x7fc3bf429700 (LWP 12134))]
#0  0x00007fc3d65d6eaf in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fc3d65d6eaf in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1  0x00007fc3d6a4896c in cherokee_bogotime_lock_read () at bogotime.c:69
#2  0x00007fc3d680368e in thread_update_bogo_now (thd=0x815b90) at thread.c:68
#3  0x00007fc3d6805ef5 in thread_update_bogo_now (thd=0x815b90) at thread.c:59
#4  watch_accept_MULTI_THREAD (fdwatch_msecs=1000, block=<optimized out>, 
    thd=0x815b90) at thread.c:1945
#5  cherokee_thread_step_MULTI_THREAD (thd=thd@entry=0x815b90, 
    dont_block=dont_block@entry=false) at thread.c:2067
#6  0x00007fc3d680621a in thread_routine (data=0x815b90) at thread.c:99
#7  0x00007fc3d65d19a7 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fc3d5a5dd0f in clone () from /lib64/libc.so.6

@skinkie skinkie self-assigned this May 9, 2018

@skinkie

This comment has been minimized.

Member

skinkie commented Jun 9, 2018

Below seems to be the interaction. @kinnison @alobbs

Thread 60 (Thread 0x7fac4dac5700 (LWP 16055)):
#0  0x00007fac6e53270c in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007fac6e52ba25 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x00007fac6e75ba3e in watch_accept_MULTI_THREAD (fdwatch_msecs=1000, block=true, thd=0x5646c259b0a0) at thread.c:1912
#3  cherokee_thread_step_MULTI_THREAD (thd=thd@entry=0x5646c259b0a0, dont_block=dont_block@entry=false) at thread.c:2067
#4  0x00007fac6e75bcba in thread_routine (data=0x5646c259b0a0) at thread.c:99
#5  0x00007fac6e52908a in start_thread () from /usr/lib/libpthread.so.0
#6  0x00007fac6ddea1bf in clone () from /usr/lib/libc.so.6

Thread 59 (Thread 0x7fac4e2c6700 (LWP 16054)):
#0  0x00007fac6e52e30a in pthread_rwlock_wrlock () from /usr/lib/libpthread.so.0
#1  0x00007fac6e99e70d in cherokee_bogotime_update () at bogotime.c:191
#2  0x00007fac6e992fef in cherokee_fdpoll_watch (fdp=<optimized out>, timeout_msecs=1000) at fdpoll.c:330
#3  0x00007fac6e75bbbf in watch_accept_MULTI_THREAD (fdwatch_msecs=<optimized out>, block=<optimized out>, thd=<optimized out>) at thread.c:1916
#4  cherokee_thread_step_MULTI_THREAD (thd=thd@entry=0x5646c25949e0, dont_block=dont_block@entry=false) at thread.c:2067
#5  0x00007fac6e75bcba in thread_routine (data=0x5646c25949e0) at thread.c:99
#6  0x00007fac6e52908a in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007fac6ddea1bf in clone () from /usr/lib/libc.so.6

Thread 58 (Thread 0x7fac4eac7700 (LWP 16053)):
#0  0x00007fac6e52de9b in pthread_rwlock_rdlock () from /usr/lib/libpthread.so.0
#1  0x00007fac6e99e5ac in cherokee_bogotime_lock_read () at bogotime.c:69
#2  0x00007fac6e7591de in thread_update_bogo_now (thd=0x5646c258e320) at thread.c:68
#3  0x00007fac6e75bb88 in thread_update_bogo_now (thd=0x5646c258e320) at thread.c:1990
#4  watch_accept_MULTI_THREAD (fdwatch_msecs=1000, block=<optimized out>, thd=0x5646c258e320) at thread.c:1945
#5  cherokee_thread_step_MULTI_THREAD (thd=thd@entry=0x5646c258e320, dont_block=dont_block@entry=false) at thread.c:2067
#6  0x00007fac6e75bcba in thread_routine (data=0x5646c258e320) at thread.c:99
#7  0x00007fac6e52908a in start_thread () from /usr/lib/libpthread.so.0
#8  0x00007fac6ddea1bf in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7fac6edce480 (LWP 15996)):
#0  0x00007fac6e52e30a in pthread_rwlock_wrlock () from /usr/lib/libpthread.so.0
#1  0x00007fac6e99e70d in cherokee_bogotime_update () at bogotime.c:191
#2  0x00007fac6e992fef in cherokee_fdpoll_watch (fdp=<optimized out>, timeout_msecs=1000) at fdpoll.c:330
#3  0x00007fac6e75bbbf in watch_accept_MULTI_THREAD (fdwatch_msecs=<optimized out>, block=<optimized out>, thd=<optimized out>) at thread.c:1916
#4  cherokee_thread_step_MULTI_THREAD (thd=0x5646c241c550, dont_block=dont_block@entry=true) at thread.c:2067
#5  0x00007fac6e755a33 in cherokee_server_step (srv=0x5646c2390660) at server.c:1157
#6  0x00005646c20c272c in main (argc=<optimized out>, argv=<optimized out>) at main_worker.c:407
@skinkie

This comment has been minimized.

Member

skinkie commented Jun 9, 2018

@skinkie

This comment has been minimized.

Member

skinkie commented Jul 9, 2018

At this moment we are evaluating this, one week and no deadlock. Must investigate why and how this happens. Help wanted.

diff --git a/cherokee/thread.c b/cherokee/thread.c
index 5bc00322..1f5cdc02 100644
--- a/cherokee/thread.c
+++ b/cherokee/thread.c
@@ -65,7 +65,7 @@ thread_update_bogo_now (cherokee_thread_t *thd)
 
        /* Update struct tm
         */
-       cherokee_bogotime_lock_read();
+       // cherokee_bogotime_lock_read();
 
        memcpy (&thd->bogo_now_tmgmt, &cherokee_bogonow_tmgmt, sizeof(struct tm));
        memcpy (&thd->bogo_now_tmloc, &cherokee_bogonow_tmloc, sizeof(struct tm));
@@ -75,7 +75,7 @@ thread_update_bogo_now (cherokee_thread_t *thd)
        cherokee_buffer_clean (&thd->bogo_now_strgmt);
        cherokee_buffer_add_buffer (&thd->bogo_now_strgmt, &cherokee_bogonow_strgmt);
 
-       cherokee_bogotime_release();
+       // cherokee_bogotime_release();
 }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment