Symbolic name registration causes application hangs

[brycelelbach]

Running Fibonacci on ariel00 with the below options and the below git hash leads to a hang:

[23:12:11]:wash@ariel00.hermione:/.../gcc-4.6.2-release:0:$ ~/install/hpx/gcc-4.6.2-release/bin/fibonacci \
    --hpx:print-counter /threads{locality#*/worker-thread#*}/idle-rate \
    --hpx:print-counter-interval 1 -t16 --n-value 20 --version

HPX - High Performance ParalleX
A general purpose parallel C++ runtime system for distributed applications
of any scale.

Copyright (c) 2007-2013 The STE||AR Group, Louisiana State University,
http://stellar.cct.lsu.edu, email:hpx-users@stellar.cct.lsu.edu

Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

Versions:
  HPX: V0.9.8-trunk (AGAS: V3.0), Git: 4fada5472f31d87b25618ca6f62f993ebc99d7b4
  Boost: V1.53.0
  Hwloc: V1.7.0

Build:
  Type: release
  Date: Dec 17 2013 22:39:44
  Platform: linux
  Compiler: GNU C++ version 4.6.4
  Standard Library: GNU libstdc++ version 20130412

A stack trace from all threads reveals that all the scheduler threads are looking for
work, except for one thread, which sits in addressing_space::register_name_async attempting
to acquire a spinlock. It consistently remains in spinlock acquisition.

Thread 24 (Thread 0x2aaab1262700 (LWP 2141)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1262700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 23 (Thread 0x2aaab1463700 (LWP 2142)):
#0  0x00002aaaae0f2f23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabb8b64b in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabd56324 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1463700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 22 (Thread 0x2aaab1664700 (LWP 2143)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1664700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 21 (Thread 0x2aaab1865700 (LWP 2144)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1865700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 20 (Thread 0x2aaab1a66700 (LWP 2145)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1a66700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 19 (Thread 0x2aaab1c67700 (LWP 2146)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1c67700) at pthread_create.c:311
#4  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 18 (Thread 0x2aaab1e68700 (LWP 2147)):
#0  __pthread_mutex_unlock_usercnt (mutex=0x2aaaaaef9200, decr=1) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab1e68700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 17 (Thread 0x2aaab2069700 (LWP 2148)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef7900) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2069700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 16 (Thread 0x2aaab226a700 (LWP 2149)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaab0fea780) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1842c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab226a700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 15 (Thread 0x2aaab246b700 (LWP 2150)):
#0  0x00002aaaabc17f67 in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab246b700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 14 (Thread 0x2aaab266c700 (LWP 2151)):
#0  0x00002aaaabc181ca in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab266c700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 13 (Thread 0x2aaab286d700 (LWP 2152)):
#0  __GI___pthread_mutex_unlock (mutex=0x2aaaaaef8300) at pthread_mutex_unlock.c:297
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1814c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab286d700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 12 (Thread 0x2aaab2a6e700 (LWP 2153)):
#0  __pthread_mutex_unlock_usercnt (mutex=0x2aaab0feb180, decr=1) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc1842c in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2a6e700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 11 (Thread 0x2aaab2c6f700 (LWP 2154)):
#0  0x00002aaaab8c2550 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2c6f700) at pthread_create.c:311
#6  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 10 (Thread 0x2aaab2e70700 (LWP 2155)):
#0  0x00002aaaab91d3c0 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#6  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab2e70700) at pthread_create.c:311
#7  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 9 (Thread 0x2aaab3071700 (LWP 2156)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef7680) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3071700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x2aaab3272700 (LWP 2157)):
#0  0x00002aaaaddf806b in __pthread_mutex_trylock (mutex=0x2aaaaaef6780) at pthread_mutex_trylock.c:64
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3272700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x2aaab3473700 (LWP 2158)):
#0  0x00002aaaab927e66 in hpx::threads::policies::thread_queue<boost::mutex>::cleanup_terminated_locked_helper(bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaab8c2606 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#6  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3473700) at pthread_create.c:311
#7  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x2aaab3674700 (LWP 2159)):
#0  0x00002aaaaddfcacd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaaba2e4e3 in hpx::util::spinlock_pool<hpx::naming::gid_type::tag>::scoped_lock::lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaba5ce2d in hpx::lcos::future<bool> hpx::agas::stubs::symbol_namespace::service_async<bool>(hpx::naming::id_type const&, hpx::agas::request const&, hpx::threads::thread_priority) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaba90047 in hpx::agas::addressing_service::register_name_async(std::string const&, hpx::naming::id_type const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaba76e77 in hpx::agas::register_name_sync(std::string const&, hpx::naming::id_type const&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabd13e5c in hpx::performance_counters::register_with_agas(hpx::lcos::future<hpx::naming::id_type>, std::string const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabd1f594 in void hpx::lcos::detail::invoke_continuation<hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::lcos::future<hpx::naming::id_type>, hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type> >(hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >&, hpx::lcos::future<hpx::naming::id_type>&, hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type>&, mpl_::bool_<false>) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabd1fbf3 in hpx::lcos::detail::continuation<hpx::lcos::future<hpx::naming::id_type>, hpx::util::detail::bound<hpx::naming::id_type (*)(hpx::lcos::future<hpx::naming::id_type>, std::string const&), hpx::util::tuple<hpx::util::detail::placeholder<1ul>, std::string, void, void, void, void, void, void> >, hpx::naming::id_type>::run(boost::intrusive_ptr<hpx::lcos::detail::future_data<hpx::naming::id_type> > const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaab99fee8 in void hpx::lcos::detail::future_data<hpx::naming::id_type>::set_result<hpx::naming::id_type>(hpx::naming::id_type&&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#9  0x00002aaaab9a00a4 in hpx::lcos::detail::promise<hpx::naming::id_type, hpx::naming::gid_type>::set_value(hpx::naming::gid_type&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#10 0x00002aaaab99e605 in void hpx::applier::detail::apply_helper<hpx::actions::direct_action1<void (hpx::lcos::base_lco_with_value<hpx::naming::gid_type, hpx::naming::gid_type>::*)(hpx::naming::gid_type&&), &hpx::lcos::base_lco_with_value<hpx::naming::gid_type, hpx::naming::gid_type>::set_value_nonvirt, hpx::actions::detail::this_type>, mpl_::bool_<true> >::call<hpx::util::tuple<hpx::naming::gid_type&&, void, void, void, void, void, void, void> >(hpx::naming::id_type const&, unsigned long, hpx::threads::thread_priority, hpx::util::tuple<hpx::naming::gid_type&&, void, void, void, void, void, void, void>&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#11 0x00002aaaab99ead8 in hpx::actions::typed_continuation<hpx::naming::gid_type>::trigger_value(hpx::naming::gid_type&&) const () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#12 0x00002aaaabb285dc in hpx::util::detail::vtable<false>::type<hpx::util::detail::bound<hpx::actions::action<hpx::components::server::runtime_support, hpx::naming::gid_type, hpx::util::tuple<hpx::performance_counters::counter_info, void, void, void, void, void, void, void>, hpx::actions::result_action1<hpx::naming::gid_type (hpx::components::server::runtime_support::*)(hpx::performance_counters::counter_info const&), &hpx::components::server::runtime_support::create_performance_counter, hpx::actions::detail::this_type> >::continuation_thread_object_function_1, hpx::util::tuple<boost::shared_ptr<hpx::actions::continuation>, hpx::naming::gid_type (hpx::components::server::runtime_support::*)(hpx::performance_counters::counter_info const&), hpx::components::server::runtime_support*, hpx::performance_counters::counter_info, void, void, void, void> >, hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), void, void>::invoke(void**, hpx::threads::thread_state_ex_enum&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#13 0x00002aaaab959469 in hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>::operator()() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#14 0x00002aaaab90f589 in void hpx::util::coroutines::detail::lx::trampoline<hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#15 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x2aaab3875700 (LWP 2160)):
#0  __pthread_mutex_trylock (mutex=0x2aaaaaef6280) at pthread_mutex_trylock.c:29
#1  0x00002aaaab927848 in boost::unique_lock<boost::mutex>::try_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2599 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3875700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x2aaab3a76700 (LWP 2161)):
#0  0x00002aaaaddf8e1f in __pthread_mutex_unlock_usercnt (mutex=0x2aaaaaef6500, decr=<optimized out>) at pthread_mutex_unlock.c:52
#1  0x00002aaaab91d3e8 in boost::unique_lock<boost::mutex>::~unique_lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab8c2623 in hpx::threads::policies::thread_queue<boost::mutex>::wait_or_add_new(unsigned long, bool, long&, unsigned long&, hpx::threads::policies::thread_queue<boost::mutex>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc181db in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#7  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3a76700) at pthread_create.c:311
#8  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x2aaab3c77700 (LWP 2162)):
#0  0x00002aaaabc18251 in void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>&, boost::atomic<hpx::state>&, long&, unsigned long&, unsigned long&, hpx::util::function_nonser<void ()> const&) [clone .constprop.3291] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#1  0x00002aaaabc71ef4 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabc721f0 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab3c77700) at pthread_create.c:311
#5  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x2aaab417a700 (LWP 2163)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabb13185 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) [clone .constprop.3950] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabb171ea in hpx::components::server::runtime_support::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaab91d75f in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait_helper(boost::mutex&, boost::condition_variable_any&, bool&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaad2f0fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaaddf5e0e in start_thread (arg=0x2aaab417a700) at pthread_create.c:311
#6  0x00002aaaae0f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x2aaaaad5b480 (LWP 2131)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabd56563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaab93ac20 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaab914755 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run(hpx::util::function_nonser<int ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaab967cec in hpx::detail::run(hpx::runtime&, hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::variables_map&, hpx::runtime_mode, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaab968d70 in hpx::detail::run_priority_local(hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::util::command_line_handling&, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaab96ae6b in hpx::run_or_start(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaab96b3a2 in hpx::init(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaaab12619 in main ()

The problem does not show up without the performance counter.

[brycelelbach]

Occassionally, if I use a performance counter with an interval, I get a similar hang at application shutdown.

[brycelelbach]

This actually affects all applications as the race condition is occasionally hit for the bootstrap barriers:

Thread 9 (Thread 0x2aaab12f4700 (LWP 4720)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab12f4700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x2aaab14f5700 (LWP 4721)):
#0  0x00002aaaae285f23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabd1e64b in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabee9324 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#4  0x00002aaaadf88e0e in start_thread (arg=0x2aaab14f5700) at pthread_create.c:311
#5  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x2aaab16f6700 (LWP 4722)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab16f6700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x2aaab18f7700 (LWP 4723)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab18f7700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x2aaab1af8700 (LWP 4724)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab1af8700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x2aaab1cf9700 (LWP 4725)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#3  0x00002aaaadf88e0e in start_thread (arg=0x2aaab1cf9700) at pthread_create.c:311
#4  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x2aaab1efa700 (LWP 4726)):
#0  0x00002aaaadf8facd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00002aaaabbc14e3 in hpx::util::spinlock_pool<hpx::naming::gid_type::tag>::scoped_lock::lock() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabbefe2d in hpx::lcos::future<bool> hpx::agas::stubs::symbol_namespace::service_async<bool>(hpx::naming::id_type const&, hpx::agas::request const&, hpx::threads::thread_priority) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabc23047 in hpx::agas::addressing_service::register_name_async(std::string const&, hpx::naming::id_type const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabc09e77 in hpx::agas::register_name_sync(std::string const&, hpx::naming::id_type const&, hpx::error_code&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabb2e94b in hpx::create_barrier(unsigned long, char const*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabb21068 in hpx::pre_main(hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabacef22 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run_helper(hpx::util::function_nonser<int ()>, int&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaabaa3343 in hpx::util::detail::vtable<false>::type<boost::_bi::bind_t<hpx::threads::detail::tagged_thread_state<hpx::threads::thread_state_enum>, boost::_mfi::mf2<hpx::threads::detail::tagged_thread_state<hpx::threads::thread_state_enum>, hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, hpx::util::function_nonser<int ()>, int&>, boost::_bi::list3<boost::_bi::value<hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<hpx::util::function_nonser<int ()> >, boost::reference_wrapper<int> > >, hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), void, void>::invoke(void**, hpx::threads::thread_state_ex_enum&&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#9  0x00002aaaabaec469 in hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>::operator()() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#10 0x00002aaaabaa2589 in void hpx::util::coroutines::detail::lx::trampoline<hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(hpx::util::coroutines::detail::coroutine_impl_wrapper<hpx::util::function_nonser<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum)>, hpx::util::coroutines::coroutine<hpx::threads::thread_state_enum (hpx::threads::thread_state_ex_enum), hpx::threads::detail::coroutine_allocator, hpx::util::coroutines::detail::lx::x86_linux_context_impl>, hpx::util::coroutines::detail::lx::x86_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#11 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x2aaab22fc700 (LWP 4727)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabca6185 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) [clone .constprop.3950] () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabcaa1ea in hpx::components::server::runtime_support::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabab075f in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait_helper(boost::mutex&, boost::condition_variable_any&, bool&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaaacc40fa in thread_proxy () from /opt/boost/1.53.0-release/stage/lib/libboost_thread.so.1.53.0
#5  0x00002aaaadf88e0e in start_thread (arg=0x2aaab22fc700) at pthread_create.c:311
#6  0x00002aaaae28593d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x2aaaaacf0b40 (LWP 4715)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaabee9563 in hpx::util::io_service_pool::thread_run(unsigned long) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#2  0x00002aaaabacdc20 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::wait() () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#3  0x00002aaaabaa7755 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::run(hpx::util::function_nonser<int ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#4  0x00002aaaabafacec in hpx::detail::run(hpx::runtime&, hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::variables_map&, hpx::runtime_mode, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#5  0x00002aaaabafbd70 in hpx::detail::run_priority_local(hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::util::command_line_handling&, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#6  0x00002aaaabafde6b in hpx::run_or_start(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode, bool) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#7  0x00002aaaabafe3a2 in hpx::init(hpx::util::function_nonser<int (boost::program_options::variables_map&)> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) () from /home/wash/install/hpx/gcc-4.6.2-release/lib/hpx/libhpx.so.0
#8  0x00002aaaaaad5d9f in main ()

[hkaiser]

Please enable lock tracking and report back whether this triggers some held lock during suspension.

[sithhell]

Please try again with the latest commit. I am still seeing occasional hangs on shutdown. Investigating.

[hkaiser]

What's the current status of this?

[sithhell]

I was not able to reproduce this specific hang. Just recently ran a job on 256 cores with around 1024 performance counter enabled.
Please reopen if the problem is still there.