Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #12085] deadlock in ido reconnect #4340

Closed
icinga-migration opened this issue Jul 1, 2016 · 3 comments

Comments

Projects
None yet
1 participant
@icinga-migration
Copy link
Member

commented Jul 1, 2016

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12085

Created by tobiasvdk on 2016-07-01 09:48:57 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2016-07-12 10:00:08 +00:00)
Target Version: 2.5.0
Last Update: 2016-07-12 10:00:08 +00:00 (in Redmine)

Icinga Version: r2.4.10-509-g208fd53
Backport?: Not yet backported
Include in Changelog: 1

My icinga 2 master hangs - __lll_lock_wait in multiple threads. This happened after an icinga reload using mariadb (10.1.14+maria-1~jessie) on debian jessie.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.4.10-509-g208fd53)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 4.6.0-0.bpo.1-amd64
  Architecture: x86_64

last log messages:

[2016-07-01 08:48:41 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9045.48/s (542729/min 923285/5min 923310/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:48:56 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9047.78/s (542867/min 1078544/5min 1078569/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:49:11 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9038.02/s (542281/min 1233021/5min 1233046/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:49:26 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 7699.67/s (461980/min 1307774/5min 1307799/15min); empty in infinite time, your database isn't able to keep up

(gdb) info threads
  Id   Target Id         Frame 
  109  Thread 0x2b51a0355700 (LWP 25360) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  108  Thread 0x2b51a0e00700 (LWP 25361) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  107  Thread 0x2b51a1001700 (LWP 25362) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  106  Thread 0x2b51a1202700 (LWP 25363) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  105  Thread 0x2b51a1403700 (LWP 25364) "icinga2" 0x00002b519acbc438 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  104  Thread 0x2b51a1604700 (LWP 25365) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  103  Thread 0x2b51a8e00700 (LWP 25414) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  102  Thread 0x2b51a9001700 (LWP 25415) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  101  Thread 0x2b51a9202700 (LWP 25416) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  100  Thread 0x2b51a9403700 (LWP 25417) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  99   Thread 0x2b51a9604700 (LWP 25418) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  98   Thread 0x2b51a9805700 (LWP 25419) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  97   Thread 0x2b51a9a06700 (LWP 25420) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  96   Thread 0x2b51a9c07700 (LWP 25421) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  95   Thread 0x2b51a9e08700 (LWP 25422) "icinga2" 0x00002b519acbc438 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  94   Thread 0x2b51cc600700 (LWP 25423) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
* 93   Thread 0x2b51cd600700 (LWP 25424) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  92   Thread 0x2b51cde00700 (LWP 25425) "icinga2" 0x00002b519acbeccd in accept () from /lib/x86_64-linux-gnu/libpthread.so.0
  91   Thread 0x2b51ce001700 (LWP 25426) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  90   Thread 0x2b51d0a00700 (LWP 25428) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  89   Thread 0x2b51d0c01700 (LWP 25429) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  88   Thread 0x2b51d0e02700 (LWP 25430) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  87   Thread 0x2b51d1003700 (LWP 25431) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  86   Thread 0x2b51d1204700 (LWP 25432) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  85   Thread 0x2b51d2200700 (LWP 25433) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  84   Thread 0x2b51d2e00700 (LWP 25434) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  83   Thread 0x2b51d3600700 (LWP 25435) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  82   Thread 0x2b51d9200700 (LWP 25436) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  81   Thread 0x2b51d9401700 (LWP 25437) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  80   Thread 0x2b51d9e01700 (LWP 25438) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  79   Thread 0x2b51d9602700 (LWP 25439) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  78   Thread 0x2b51da002700 (LWP 25440) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  77   Thread 0x2b51da203700 (LWP 25442) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  76   Thread 0x2b51f5600700 (LWP 25444) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  75   Thread 0x2b520ca00700 (LWP 25446) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  74   Thread 0x2b522e200700 (LWP 25448) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  73   Thread 0x2b526c200700 (LWP 25455) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  72   Thread 0x2b526c602700 (LWP 25457) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  71   Thread 0x2b526c401700 (LWP 25459) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  70   Thread 0x2b51ce202700 (LWP 25460) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  69   Thread 0x2b526ce00700 (LWP 25461) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  68   Thread 0x2b526d801700 (LWP 25463) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  67   Thread 0x2b526d600700 (LWP 25464) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  66   Thread 0x2b526da02700 (LWP 25465) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  65   Thread 0x2b526e401700 (LWP 25467) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  64   Thread 0x2b526e200700 (LWP 25474) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  63   Thread 0x2b526e602700 (LWP 25475) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  62   Thread 0x2b526e803700 (LWP 25476) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  61   Thread 0x2b526ea04700 (LWP 25477) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  60   Thread 0x2b526ec05700 (LWP 25478) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  59   Thread 0x2b526ee06700 (LWP 25479) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  58   Thread 0x2b526f208700 (LWP 25482) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  57   Thread 0x2b526f007700 (LWP 25485) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  56   Thread 0x2b526f60a700 (LWP 25488) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  55   Thread 0x2b5270801700 (LWP 25490) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  54   Thread 0x2b5270600700 (LWP 25496) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  53   Thread 0x2b5270a02700 (LWP 25505) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  52   Thread 0x2b5271200700 (LWP 25508) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  51   Thread 0x2b526f409700 (LWP 25509) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  50   Thread 0x2b5271e00700 (LWP 25520) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  49   Thread 0x2b5273001700 (LWP 25528) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  48   Thread 0x2b5272e00700 (LWP 25531) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  47   Thread 0x2b5287a00700 (LWP 25535) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  46   Thread 0x2b5273202700 (LWP 25547) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  45   Thread 0x2b5305200700 (LWP 25548) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  44   Thread 0x2b5315600700 (LWP 25552) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  43   Thread 0x2b5315801700 (LWP 25553) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  42   Thread 0x2b5315a02700 (LWP 25569) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  41   Thread 0x2b5316e00700 (LWP 25573) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  40   Thread 0x2b5317001700 (LWP 25578) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  39   Thread 0x2b5317202700 (LWP 25579) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  38   Thread 0x2b5317604700 (LWP 25610) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  37   Thread 0x2b5317403700 (LWP 25614) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  36   Thread 0x2b5317805700 (LWP 25615) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  35   Thread 0x2b5317a06700 (LWP 25616) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  34   Thread 0x2b5317c07700 (LWP 25620) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  33   Thread 0x2b5318009700 (LWP 25622) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  32   Thread 0x2b5317e08700 (LWP 25627) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  31   Thread 0x2b531820a700 (LWP 25628) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  30   Thread 0x2b531840b700 (LWP 25632) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  29   Thread 0x2b531860c700 (LWP 25633) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  28   Thread 0x2b531880d700 (LWP 25634) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  27   Thread 0x2b5318a0e700 (LWP 25635) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  26   Thread 0x2b5318c0f700 (LWP 25646) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  25   Thread 0x2b5318e10700 (LWP 25649) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  24   Thread 0x2b531a600700 (LWP 25655) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  23   Thread 0x2b531ae00700 (LWP 25659) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  22   Thread 0x2b531c600700 (LWP 25678) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  21   Thread 0x2b531d600700 (LWP 25692) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  20   Thread 0x2b531d801700 (LWP 25699) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  19   Thread 0x2b531da02700 (LWP 25703) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  18   Thread 0x2b531dc03700 (LWP 25704) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  17   Thread 0x2b531de04700 (LWP 25721) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  16   Thread 0x2b531e005700 (LWP 25731) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  15   Thread 0x2b531e206700 (LWP 25736) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  14   Thread 0x2b531e407700 (LWP 25740) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  13   Thread 0x2b531e608700 (LWP 25773) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  12   Thread 0x2b531e809700 (LWP 25775) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  11   Thread 0x2b531ea0a700 (LWP 25778) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  10   Thread 0x2b531ec0b700 (LWP 25781) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  9    Thread 0x2b531ee0c700 (LWP 25784) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  8    Thread 0x2b531f00d700 (LWP 25787) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  7    Thread 0x2b531f20e700 (LWP 25867) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  6    Thread 0x2b531f40f700 (LWP 25895) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  5    Thread 0x2b531f610700 (LWP 25907) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x2b531f811700 (LWP 26006) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x2b531fa12700 (LWP 26013) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x2b531fc13700 (LWP 26324) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  1    Thread 0x2b5199edef00 (LWP 25359) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) thread 93
[Switching to thread 93 (Thread 0x2b51cd600700 (LWP 25424))]

#9  0x00002b51a4ed83a5 in icinga::DbConnection::UpdateObject (this=this@entry=0x2b51aae60c00, object=(boost::intrusive_ptr) 0x2b51bb17d600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:399
399 dbobj->SendConfigUpdate();
(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a4ef8935 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51b71b72e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a4ed6249 in Lock (this=0x2b51cd5fdd50) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b51cd5fdd50) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::DbObject::SendVarsConfigUpdate (this=this@entry=0x2b5202c642e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbobject.cpp:164
#8  0x00002b51a4ed7bec in icinga::DbObject::SendConfigUpdate (this=0x2b5202c642e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbobject.cpp:86
#9  0x00002b51a4ed83a5 in icinga::DbConnection::UpdateObject (this=this@entry=0x2b51aae60c00, object=(boost::intrusive_ptr) 0x2b51bb17d600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:399

#10 0x00002b51a4ed87d5 in icinga::DbConnection::UpdateAllObjects (this=this@entry=0x2b51aae60c00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:415
#11 0x00002b51a464ed68 in icinga::IdoMysqlConnection::Reconnect (this=0x2b51aae60c00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido_mysql/idomysqlconnection.cpp:413
#12 0x00002b519afcfd63 in operator() (this=0x2b51cd5ffc50) at /usr/include/boost/function/function_template.hpp:767
#13 icinga::WorkQueue::WorkerThreadProc (this=0x2b51aae60ea8) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/workqueue.cpp:228
#14 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#15 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread 106 [Switching to thread 106 (Thread 0x2b51a1202700 (LWP 25363))]
#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a3ad7552 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51bc059600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a3a46957 in Lock (this=0x2b51a1200780) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b51a1200780) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::CIB::CalculateServiceCheckStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:137
#8  0x00002b51a3a9a391 in icinga::CIB::StatsFunc (status=(boost::intrusive_ptr) 0x2b51c9889380, perfdata=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:292
#9  0x00002b519afaa6aa in operator() (a1=(boost::intrusive_ptr) 0x1, a0=(boost::intrusive_ptr) 0x2b51c9889380, this=0x2b51a08b6ad8)

at /usr/include/boost/function/function_template.hpp:767

#10 icinga::StatsFunction::Invoke (this=this@entry=0x2b51a08b6ac0, status=(boost::intrusive_ptr) 0x2b51c9889380, perfdata=(boost::intrusive_ptr) 0x2b51c988a280)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/statsfunction.cpp:32

#11 0x00002b51a3a46ebb in icinga::CIB::GetFeatureStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:260
#12 0x00002b51a3e67f78 in icinga::ClusterCheckTask::ScriptFunc (checkable=(boost::intrusive_ptr) 0x2b51b5967d80, cr=(boost::intrusive_ptr) 0x2b51c9845080, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=useResolvedMacros@entry=false) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/methods/clusterchecktask.cpp:59

#13 0x00002b51a3e70b00 in icinga::FunctionWrapperV const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool>

(function=
0x2b51a3e67ed0 <icinga::ClusterCheckTask::ScriptFunc(boost::intrusive_ptricinga::Checkable const&, boost::intrusive_ptricinga::CheckResult const&, boost::intrusive_ptricinga::Dictionary const&, bool)>, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/functionwrapper.hpp:171

#14 0x00002b51a3e6c42d in operator()&, const boost::intrusive_ptr&, const boost::intrusive_ptr&, bool), const std::vector&), boost::_bi::list1&> > (f=, a=, this=)

at /usr/include/boost/bind/bind.hpp:303

#15 operator() > (a1=..., this=) at /usr/include/boost/bind/bind_template.hpp:47
#16 boost::detail::function::function_obj_invoker1 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool), std::vector > const&), boost::_bi::list2 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool)>, boost::arg<1> > >, icinga::Value, std::vector > const&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#17 0x00002b519afaa61d in operator() (a0=std::vector of length 4, capacity 4 = {...}, this=) at /usr/include/boost/function/function_template.hpp:767
#18 icinga::Function::Invoke (this=, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/function.cpp:34
#19 0x00002b51a3a2b44b in icinga::CheckCommand::Execute (this=0x2b51ab419640, checkable=..., cr=(boost::intrusive_ptr) 0x2b51c9845080, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkcommand.cpp:36

#20 0x00002b51a3ab1dff in icinga::Checkable::ExecuteCheck (this=0x2b51b5967d80) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkable-check.cpp:448
#21 0x00002b51a40bc0bb in icinga::CheckerComponent::ExecuteCheckHelper (this=0x2b51ab4a8180, checkable=(boost::intrusive_ptr) 0x2b51b5967d80)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/checker/checkercomponent.cpp:204

#22 0x00002b519afe0187 in operator() (this=0x2b51a12019d0) at /usr/include/boost/function/function_template.hpp:767
#23 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dc2b0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#24 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#25 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#26 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread 72 [Switching to thread 72 (Thread 0x2b526c602700 (LWP 25457))]
#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b519b01dfd5 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51bc059600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b519afa717e in Lock (this=0x2b526c6014b0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=0x2b51bc059600, this=0x2b526c6014b0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:56
#7  icinga::ConfigObject::SetAuthority (this=0x2b51bc059600, authority=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/configobject.cpp:468
#8  0x00002b519b9b29c6 in icinga::ApiListener::UpdateObjectAuthority () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/remote/authority.cpp:78
#9  0x00002b519b05999c in operator() (a0=..., this=) at /usr/include/boost/function/function_template.hpp:767
#10 m_invoke&)>, 0u, const boost::intrusive_ptr&> (args=..., func=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:92

#11 operator()&)>, const boost::intrusive_ptr&, 1ul> (args=..., func=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:81

#12 m_invoke >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (connectionBody=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:117

#13 operator() >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (connectionBody=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:110

#14 dereference (this=0x2b526c601720) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
#15 dereference&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (f=...)

at /usr/include/boost/iterator/iterator_facade.hpp:514

#16 operator* (this=0x2b526c601720) at /usr/include/boost/iterator/iterator_facade.hpp:639
#17 operator()&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (last=..., 

first=..., this=) at /usr/include/boost/signals2/optional_last_value.hpp:55

#18 operator(), boost::signals2::detail::slot_call_iterator_t&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (last=..., first=..., combiner=..., this=) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
#19 boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (this=, args#0=(boost::intrusive_ptr) 0x2b51a085ebf0) at /usr/include/boost/signals2/detail/signal_template.hpp:246
#20 0x00002b519b000bd9 in operator() (args#0=(boost::intrusive_ptr) 0x2b51a085ebf0, this=0x2b51a085ec08) at /usr/include/boost/signals2/detail/signal_template.hpp:695
#21 icinga::Timer::Call (this=0x2b51a085ebf0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/timer.cpp:94
#22 0x00002b519afe0187 in operator() (this=0x2b526c6019d0) at /usr/include/boost/function/function_template.hpp:767
#23 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dbfb0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#24 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#25 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#26 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread 71 [Switching to thread 71 (Thread 0x2b526c401700 (LWP 25459))]
#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a3ad7552 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51b595ef00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a3a46957 in Lock (this=0x2b526c3ff810) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b526c3ff810) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::CIB::CalculateServiceCheckStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:137
#8  0x00002b51a3e62da6 in icinga::IcingaCheckTask::ScriptFunc (service=(boost::intrusive_ptr) 0x2b51b5d62600, cr=(boost::intrusive_ptr) 0x2b526a678000, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=useResolvedMacros@entry=false) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/methods/icingachecktask.cpp:66

#9  0x00002b51a3e70b00 in icinga::FunctionWrapperV const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool>

(function=
0x2b51a3e61bf0 <icinga::IcingaCheckTask::ScriptFunc(boost::intrusive_ptricinga::Checkable const&, boost::intrusive_ptricinga::CheckResult const&, boost::intrusive_ptricinga::Dictionary const&, bool)>, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/functionwrapper.hpp:171

#10 0x00002b51a3e6c42d in operator()&, const boost::intrusive_ptr&, const boost::intrusive_ptr&, bool), const std::vector&), boost::_bi::list1&> > (f=, a=, this=)

at /usr/include/boost/bind/bind.hpp:303

#11 operator() > (a1=..., this=) at /usr/include/boost/bind/bind_template.hpp:47
#12 boost::detail::function::function_obj_invoker1 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool), std::vector > const&), boost::_bi::list2 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool)>, boost::arg<1> > >, icinga::Value, std::vector > const&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#13 0x00002b519afaa61d in operator() (a0=std::vector of length 4, capacity 4 = {...}, this=) at /usr/include/boost/function/function_template.hpp:767
#14 icinga::Function::Invoke (this=, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/function.cpp:34
#15 0x00002b51a3a2b44b in icinga::CheckCommand::Execute (this=0x2b51aa863280, checkable=..., cr=(boost::intrusive_ptr) 0x2b526a678000, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkcommand.cpp:36

#16 0x00002b51a3ab1dff in icinga::Checkable::ExecuteCheck (this=0x2b51b5d62600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkable-check.cpp:448
#17 0x00002b51a40bc0bb in icinga::CheckerComponent::ExecuteCheckHelper (this=0x2b51ab4a8180, checkable=(boost::intrusive_ptr) 0x2b51b5d62600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/checker/checkercomponent.cpp:204

#18 0x00002b519afe0187 in operator() (this=0x2b526c4009d0) at /usr/include/boost/function/function_template.hpp:767
#19 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dbfd0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#20 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#21 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#22 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Attachments

Changesets

2016-07-12 09:44:36 +00:00 by gbeutner ea57f00

Fix deadlock in DbObject::SendVars{Config,Status}Update

fixes #12127
fixes #12085

Relations:

@icinga-migration

This comment has been minimized.

Copy link
Member Author

commented Jul 12, 2016

Updated by gbeutner on 2016-07-12 09:45:57 +00:00

  • Category set to DB IDO
  • Status changed from New to Assigned
  • Assigned to set to gbeutner
  • Target Version set to 2.5.0
@icinga-migration

This comment has been minimized.

Copy link
Member Author

commented Jul 12, 2016

Updated by gbeutner on 2016-07-12 09:46:10 +00:00

  • Relates set to 12127
@icinga-migration

This comment has been minimized.

Copy link
Member Author

commented Jul 12, 2016

Updated by gbeutner on 2016-07-12 10:00:08 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset ea57f00.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.