Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition at shutdown #894

Closed
brycelelbach opened this issue Sep 27, 2013 · 2 comments
Closed

Race condition at shutdown #894

brycelelbach opened this issue Sep 27, 2013 · 2 comments
Assignees
@hkaiser
Labels
category: threadmanager difficulty: medium project: XPRESS type: defect type: race condition
Milestone
0.9.7

Comments

@brycelelbach
Copy link
Member

Sameer initial ran into this. When using perf counters with intervals, occasionally
applications hang at shutdown. Sameer ran into this specifically with jacobi. If you
remove the counter interval, the problem goes away.

While trying to reproduce this, I ran into:

[11:36:34]:wash@ariel00.hermione:/srv/scratch/wash/hpx/gcc-4.6.2-debug:0:$ ~/install/hpx/gcc-4.6.2-debug/bin/jacobi --hpx:print-counter /threadqueue{locality#*/total}/length --hpx:print-counter-interval 10 -t16 --nx 100 --ny 100 --hpx:debug-hpx-log='/tmp/foobar'
/threadqueue{locality#0/total}/length,1,0.057813,[s],0
/threadqueue{locality#0/total}/length,2,0.069163,[s],0
/threadqueue{locality#0/total}/length,3,0.080520,[s],0
/threadqueue{locality#0/total}/length,4,0.092200,[s],0
/threadqueue{locality#0/total}/length,5,0.103634,[s],0
/threadqueue{locality#0/total}/length,6,0.116258,[s],0
/threadqueue{locality#0/total}/length,7,0.127839,[s],4
/threadqueue{locality#0/total}/length,8,0.140702,[s],6
/threadqueue{locality#0/total}/length,9,0.153713,[s],0
/threadqueue{locality#0/total}/length,10,0.166569,[s],15
/threadqueue{locality#0/total}/length,11,0.183908,[s],188
/threadqueue{locality#0/total}/length,12,0.203572,[s],296
/threadqueue{locality#0/total}/length,13,0.219059,[s],467
/threadqueue{locality#0/total}/length,14,0.239922,[s],805
/threadqueue{locality#0/total}/length,15,0.254971,[s],985
/threadqueue{locality#0/total}/length,16,0.274396,[s],1075
/threadqueue{locality#0/total}/length,17,0.316999,[s],1195
/threadqueue{locality#0/total}/length,18,0.336190,[s],1287
/threadqueue{locality#0/total}/length,19,0.361825,[s],1134
/threadqueue{locality#0/total}/length,20,0.375448,[s],1144
/threadqueue{locality#0/total}/length,21,0.388000,[s],1128
/threadqueue{locality#0/total}/length,22,0.401783,[s],1141
/threadqueue{locality#0/total}/length,23,0.415038,[s],1130
/threadqueue{locality#0/total}/length,24,0.433257,[s],1080
/threadqueue{locality#0/total}/length,25,0.452430,[s],864
/threadqueue{locality#0/total}/length,26,0.470491,[s],711
/threadqueue{locality#0/total}/length,27,0.483367,[s],715
/threadqueue{locality#0/total}/length,28,0.495941,[s],585
/threadqueue{locality#0/total}/length,29,0.508105,[s],405
/threadqueue{locality#0/total}/length,30,0.520677,[s],271
/threadqueue{locality#0/total}/length,31,0.537564,[s],28
/threadqueue{locality#0/total}/length,32,0.549851,[s],28
/threadqueue{locality#0/total}/length,33,0.567388,[s],209
/threadqueue{locality#0/total}/length,34,0.599767,[s],566
/threadqueue{locality#0/total}/length,35,0.619755,[s],671
/threadqueue{locality#0/total}/length,36,0.650438,[s],817
/threadqueue{locality#0/total}/length,37,0.669644,[s],805
/threadqueue{locality#0/total}/length,38,0.682192,[s],844
/threadqueue{locality#0/total}/length,39,0.708019,[s],828
/threadqueue{locality#0/total}/length,40,0.720625,[s],777
/threadqueue{locality#0/total}/length,41,0.733635,[s],756
/threadqueue{locality#0/total}/length,42,0.749302,[s],811
/threadqueue{locality#0/total}/length,43,0.764035,[s],843
/threadqueue{locality#0/total}/length,44,0.777331,[s],788
/threadqueue{locality#0/total}/length,45,0.789402,[s],672
/threadqueue{locality#0/total}/length,46,0.802306,[s],611
/threadqueue{locality#0/total}/length,47,0.814833,[s],510
/threadqueue{locality#0/total}/length,48,0.843142,[s],483
/threadqueue{locality#0/total}/length,49,0.886697,[s],263
/threadqueue{locality#0/total}/length,50,0.900186,[s],115
/threadqueue{locality#0/total}/length,51,0.912219,[s],8
/threadqueue{locality#0/total}/length,52,0.927537,[s],30
/threadqueue{locality#0/total}/length,53,0.940744,[s],185
/threadqueue{locality#0/total}/length,54,0.959639,[s],441
/threadqueue{locality#0/total}/length,55,0.980662,[s],680
/threadqueue{locality#0/total}/length,56,0.995418,[s],673
/threadqueue{locality#0/total}/length,57,1.025120,[s],684
/threadqueue{locality#0/total}/length,58,1.043148,[s],754
/threadqueue{locality#0/total}/length,59,1.070723,[s],845
/threadqueue{locality#0/total}/length,60,1.083731,[s],910
/threadqueue{locality#0/total}/length,61,1.106773,[s],916
/threadqueue{locality#0/total}/length,62,1.120259,[s],837
/threadqueue{locality#0/total}/length,63,1.132974,[s],886
/threadqueue{locality#0/total}/length,64,1.151268,[s],913
/threadqueue{locality#0/total}/length,65,1.167832,[s],888
/threadqueue{locality#0/total}/length,66,1.180117,[s],932
/threadqueue{locality#0/total}/length,67,1.192266,[s],938
/threadqueue{locality#0/total}/length,68,1.204830,[s],834
/threadqueue{locality#0/total}/length,69,1.217442,[s],700
/threadqueue{locality#0/total}/length,70,1.230078,[s],532
/threadqueue{locality#0/total}/length,71,1.242386,[s],402
/threadqueue{locality#0/total}/length,72,1.254764,[s],303
/threadqueue{locality#0/total}/length,73,1.268161,[s],213
/threadqueue{locality#0/total}/length,74,1.280104,[s],89
/threadqueue{locality#0/total}/length,75,1.292090,[s],0
/threadqueue{locality#0/total}/length,76,1.304745,[s],21
/threadqueue{locality#0/total}/length,77,1.329637,[s],362
/threadqueue{locality#0/total}/length,78,1.346404,[s],617
/threadqueue{locality#0/total}/length,79,1.524646,[s],1565
/threadqueue{locality#0/total}/length,80,1.537759,[s],1531
/threadqueue{locality#0/total}/length,81,1.549898,[s],1420
/threadqueue{locality#0/total}/length,82,1.563930,[s],1240
/threadqueue{locality#0/total}/length,83,1.576507,[s],1082
/threadqueue{locality#0/total}/length,84,1.589651,[s],1025
/threadqueue{locality#0/total}/length,85,1.602611,[s],973
/threadqueue{locality#0/total}/length,86,1.615394,[s],850
/threadqueue{locality#0/total}/length,87,1.627921,[s],638
/threadqueue{locality#0/total}/length,88,1.639861,[s],462
/threadqueue{locality#0/total}/length,89,1.651865,[s],259
/threadqueue{locality#0/total}/length,90,1.663977,[s],157
/threadqueue{locality#0/total}/length,91,1.676160,[s],15
/threadqueue{locality#0/total}/length,92,1.688676,[s],30
/threadqueue{locality#0/total}/length,93,1.706919,[s],267
/threadqueue{locality#0/total}/length,94,1.718367,[s],427
/threadqueue{locality#0/total}/length,95,1.733932,[s],519
/threadqueue{locality#0/total}/length,96,1.763881,[s],795
/threadqueue{locality#0/total}/length,97,1.794631,[s],1119
/threadqueue{locality#0/total}/length,98,1.852072,[s],1341
/threadqueue{locality#0/total}/length,99,1.868972,[s],1410
/threadqueue{locality#0/total}/length,100,1.888228,[s],1375
/threadqueue{locality#0/total}/length,101,1.926765,[s],1443
/threadqueue{locality#0/total}/length,102,1.939151,[s],1407
/threadqueue{locality#0/total}/length,103,1.951660,[s],1348
/threadqueue{locality#0/total}/length,104,1.964307,[s],1310
/threadqueue{locality#0/total}/length,105,1.977000,[s],1155
/threadqueue{locality#0/total}/length,106,1.992746,[s],902
/threadqueue{locality#0/total}/length,107,2.004902,[s],692
/threadqueue{locality#0/total}/length,108,2.018192,[s],504
/threadqueue{locality#0/total}/length,109,2.030778,[s],337
/threadqueue{locality#0/total}/length,110,2.043093,[s],157
/threadqueue{locality#0/total}/length,111,2.055112,[s],10
/threadqueue{locality#0/total}/length,112,2.066945,[s],47
/threadqueue{locality#0/total}/length,113,2.092539,[s],319
/threadqueue{locality#0/total}/length,114,2.113392,[s],572
/threadqueue{locality#0/total}/length,115,2.148198,[s],519
/threadqueue{locality#0/total}/length,116,2.196307,[s],662
/threadqueue{locality#0/total}/length,117,2.224641,[s],958
/threadqueue{locality#0/total}/length,118,2.237482,[s],1010
/threadqueue{locality#0/total}/length,119,2.253141,[s],976
/threadqueue{locality#0/total}/length,120,2.265534,[s],919
/threadqueue{locality#0/total}/length,121,2.277865,[s],920
/threadqueue{locality#0/total}/length,122,2.292616,[s],879
/threadqueue{locality#0/total}/length,123,2.307451,[s],846
/threadqueue{locality#0/total}/length,124,2.319521,[s],840
/threadqueue{locality#0/total}/length,125,2.333801,[s],684
/threadqueue{locality#0/total}/length,126,2.347149,[s],611
/threadqueue{locality#0/total}/length,127,2.365178,[s],576
/threadqueue{locality#0/total}/length,128,2.422996,[s],188
/threadqueue{locality#0/total}/length,129,2.436127,[s],21
/threadqueue{locality#0/total}/length,130,2.456239,[s],117
/threadqueue{locality#0/total}/length,131,2.473063,[s],236
/threadqueue{locality#0/total}/length,132,2.490865,[s],430
/threadqueue{locality#0/total}/length,133,2.512201,[s],554
/threadqueue{locality#0/total}/length,134,2.562788,[s],802
/threadqueue{locality#0/total}/length,135,2.580905,[s],803
/threadqueue{locality#0/total}/length,136,2.612951,[s],826
/threadqueue{locality#0/total}/length,137,2.626438,[s],737
/threadqueue{locality#0/total}/length,138,2.639359,[s],694
/threadqueue{locality#0/total}/length,139,2.652266,[s],724
/threadqueue{locality#0/total}/length,140,2.664968,[s],550
/threadqueue{locality#0/total}/length,141,2.677006,[s],355
/threadqueue{locality#0/total}/length,142,2.691321,[s],282
/threadqueue{locality#0/total}/length,143,2.705762,[s],286
/threadqueue{locality#0/total}/length,144,2.738461,[s],184
/threadqueue{locality#0/total}/length,145,2.758408,[s],352
/threadqueue{locality#0/total}/length,146,2.772511,[s],335
/threadqueue{locality#0/total}/length,147,2.790031,[s],167
/threadqueue{locality#0/total}/length,148,2.802457,[s],60
/threadqueue{locality#0/total}/length,149,2.814306,[s],0
/threadqueue{locality#0/total}/length,150,2.826177,[s],86
/threadqueue{locality#0/total}/length,151,2.846925,[s],304
/threadqueue{locality#0/total}/length,152,2.884310,[s],695
/threadqueue{locality#0/total}/length,153,2.902303,[s],692
/threadqueue{locality#0/total}/length,154,2.951843,[s],735
/threadqueue{locality#0/total}/length,155,2.983704,[s],609
/threadqueue{locality#0/total}/length,156,2.996253,[s],609
/threadqueue{locality#0/total}/length,157,3.010261,[s],595
/threadqueue{locality#0/total}/length,158,3.021584,[s],528
/threadqueue{locality#0/total}/length,159,3.034350,[s],464
/threadqueue{locality#0/total}/length,160,3.047527,[s],350
/threadqueue{locality#0/total}/length,161,3.060393,[s],272
/threadqueue{locality#0/total}/length,162,3.075251,[s],361
/threadqueue{locality#0/total}/length,163,3.100293,[s],596
/threadqueue{locality#0/total}/length,164,3.117238,[s],538
/threadqueue{locality#0/total}/length,165,3.130154,[s],387
/threadqueue{locality#0/total}/length,166,3.144063,[s],307
/threadqueue{locality#0/total}/length,167,3.156144,[s],267
/threadqueue{locality#0/total}/length,168,3.168601,[s],122
/threadqueue{locality#0/total}/length,169,3.180180,[s],22
/threadqueue{locality#0/total}/length,170,3.192659,[s],0
/threadqueue{locality#0/total}/length,171,3.210112,[s],147
/threadqueue{locality#0/total}/length,172,3.262345,[s],720
/threadqueue{locality#0/total}/length,173,3.306261,[s],809
/threadqueue{locality#0/total}/length,174,3.333300,[s],878
/threadqueue{locality#0/total}/length,175,3.362437,[s],1067
/threadqueue{locality#0/total}/length,176,3.375277,[s],1107
/threadqueue{locality#0/total}/length,177,3.392007,[s],1134
/threadqueue{locality#0/total}/length,178,3.405456,[s],1113
/threadqueue{locality#0/total}/length,179,3.423853,[s],951
/threadqueue{locality#0/total}/length,180,3.495884,[s],608
/threadqueue{locality#0/total}/length,181,3.509205,[s],495
/threadqueue{locality#0/total}/length,182,3.521515,[s],424
/threadqueue{locality#0/total}/length,183,3.535017,[s],317
/threadqueue{locality#0/total}/length,184,3.546451,[s],160
/threadqueue{locality#0/total}/length,185,3.560973,[s],8
/threadqueue{locality#0/total}/length,186,3.573426,[s],11
/threadqueue{locality#0/total}/length,187,3.593224,[s],219
/threadqueue{locality#0/total}/length,188,3.610533,[s],434
/threadqueue{locality#0/total}/length,189,3.642887,[s],823
/threadqueue{locality#0/total}/length,190,3.666908,[s],865
/threadqueue{locality#0/total}/length,191,3.699902,[s],935
/threadqueue{locality#0/total}/length,192,3.729209,[s],1140
/threadqueue{locality#0/total}/length,193,3.756062,[s],1330
/threadqueue{locality#0/total}/length,194,3.839158,[s],910
/threadqueue{locality#0/total}/length,195,3.872213,[s],728
/threadqueue{locality#0/total}/length,196,3.884270,[s],652
/threadqueue{locality#0/total}/length,197,3.898152,[s],532
/threadqueue{locality#0/total}/length,198,3.911764,[s],382
/threadqueue{locality#0/total}/length,199,3.926193,[s],209
/threadqueue{locality#0/total}/length,200,3.938400,[s],57
/threadqueue{locality#0/total}/length,201,3.949628,[s],0
100x100 0.0253616 MLUPS
/threadqueue{locality#0/total}/length,202,3.952730,[s],0

{stack-trace}: 18 frames:
0x7f7d5dd9b6c0  : hpx::util::backtrace::backtrace(unsigned long) + 0x80 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9b7ae  : hpx::util::trace_on_new_stack() + 0x1e in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd97fdc  : hpx::detail::backtrace() + 0x18 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9bb6d  : boost::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x46 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd9c123  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::string const&, std::string const&, long) + 0x38 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd985be  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x238 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5dd98386  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x0 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5fda390a  : boost::any::any() + 0x0 in /home/wash/install/hpx/gcc-4.6.2-debug/bin/jacobi
0x7f7d5e022378  : hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc_impl(unsigned long) + 0x136 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e0219dd  : hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>::tfunc(unsigned long, hpx::threads::topology const&) + 0x4f7 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037f9f  : boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>::operator()(hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*, unsigned long, hpx::threads::topology const&) const + 0x6b in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037e28  : void boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> >::operator()<boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>&, boost::_bi::list0&, int) + 0x88 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037c93  : boost::_bi::bind_t<void, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> > >::operator()() + 0x39 in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5e037b58  : boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf2<void, hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>, unsigned long, hpx::threads::topology const&>, boost::_bi::list3<boost::_bi::value<hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex>, hpx::threads::policies::callback_notifier>*>, boost::_bi::value<unsigned long>, boost::reference_wrapper<hpx::threads::topology const> > > >::run() + 0x1e in /home/wash/install/hpx/gcc-4.6.2-debug/lib/hpx/libhpxd.so.1
0x7f7d5ac23ceb  : ??? + 0x7f7d5ac23ceb in /opt/boost/1.52.0-debug/stage/lib/libboost_thread.so.1.52.0
0x7f7d591ffe0e  : ??? + 0x7f7d591ffe0e in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f7d58f3493d  : clone + 0x6d in /lib/x86_64-linux-gnu/libc.so.6
{env}: 37 entries:
  CPATH=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/tbb/include
  DISPLAY=localhost:10.0
  EDITOR=vim
  GLIBCPP_FORCE_NEW=1
  GLIBCXX_FORCE_NEW=1
  HISTCONTROL=ignoredups
  HOME=/home/wash
  INCLUDE=/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include:/opt/intel/composer_xe_2013.1.117/mkl/include
  INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses:/opt/intel/composer_xe_2013.1.117/licenses:/opt/intel/licenses:/home/wash/intel/licenses
  IPPROOT=/opt/intel/composer_xe_2013.1.117/ipp
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/visit/2.5.2/2.5.2/linux-x86-64/lib
  LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64
  LOGNAME=wash
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MAIL=/var/mail/wash
  MANPATH=/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/opt/intel/composer_xe_2013.1.117/man/en_US:/usr/local/man:/usr/local/share/man:/usr/share/man:::/opt/intel/vtune_amplifier_xe_2013/man
  MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/compiler/lib/mic:/opt/intel/composer_xe_2013.1.117/mkl/lib/mic:/opt/intel/composer_xe_2013.1.117/tbb/lib/mic
  MKLROOT=/opt/intel/composer_xe_2013.1.117/mkl
  NLSPATH=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013.1.117/debugger/intel64/locale/%l_%t/%N
  OLDPWD=/srv/scratch/wash
  PATH=/opt/intel/vtune_amplifier_xe_2013/bin64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64:/opt/intel/composer_xe_2013.1.117/bin/intel64_mic:/opt/intel/composer_xe_2013.1.117/debugger/gui/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/visit/2.5.2/bin
  PWD=/srv/scratch/wash/hpx/gcc-4.6.2-debug
  SHELL=/bin/bash
  SHLVL=1
  SHOME=/srv/scratch/wash
  SILO_ROOT=/opt/silo/4.8-release
  SSH_CLIENT=10.1.1.1 56765 22
  SSH_CONNECTION=10.1.1.1 56765 10.1.1.13 22
  SSH_TTY=/dev/pts/0
  TBBROOT=/opt/intel/composer_xe_2013.1.117/tbb
  TERM=xterm
  USER=wash
  VISIT=/opt/visit/2.5.2/2.5.2/linux-x86-64
  VISITPLUGINDIR=/opt/visit/2.5.2/2.5.2/linux-x86-64/plugins
  VTUNE_AMPLIFIER_XE_2013_DIR=/opt/intel/vtune_amplifier_xe_2013
  _=/home/wash/install/hpx/gcc-4.6.2-debug/bin/jacobi
{locality-id}: 0
{process-id}: 11229
{function}: void hpx::threads::threadmanager_impl<SchedulingPolicy, NotificationPolicy>::tfunc_impl(std::size_t) [with SchedulingPolicy = hpx::threads::policies::local_priority_queue_scheduler<>, NotificationPolicy = hpx::threads::policies::callback_notifier, std::size_t = long unsigned int]
{file}: /home/wash/hpx/src/runtime/threads/threadmanager.cpp
{line}: 1318
{os-thread}: worker-thread#0
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_SHMEM=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=ON
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_LIMIT=5
  HPX_ACTION_ARGUMENT_LIMIT=4
  HPX_COMPONENT_CREATE_ARGUMENT_LIMIT=5
  HPX_FUNCTION_ARGUMENT_LIMIT=7
  HPX_LOCK_LIMIT=5
  HPX_TUPLE_LIMIT=10
  HPX_WAIT_ARGUMENT_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX=/home/wash/install/hpx/gcc-4.6.2-debug
{version}: V1.0.0-trunk (AGAS: V3.0), Git: d124c2eab3b774bdbd21752fd0895c314e145c60
{boost}: V1.52.0
{build-type}: debug
{date}: Sep 27 2013 10:53:42
{platform}: linux
{compiler}: GNU C++ version 4.6.4
{stdlib}: GNU libstdc++ version 20130412
{what}: assertion '!scheduler_.get_thread_count( unknown, thread_priority_default, num_thread)' failed: HPX(assertion_failure)

Aborted
@ghost ghost assigned hkaiser Sep 27, 2013
hkaiser added a commit that referenced this issue Sep 27, 2013
@hkaiser
Attempting to solve #894: Race condition at shutdown
1fb8e08
hkaiser added a commit that referenced this issue Sep 27, 2013
@hkaiser
More attempts at resolving #894: Race condition at shutdown
49f4678
@brycelelbach
Copy link
Member Author

Hartmut: I'll test tonight to see if your second commit fixes this.

hkaiser added a commit that referenced this issue Sep 28, 2013
@hkaiser
More work on #894: Race condition at shutdown
02ca1be
@hkaiser
Copy link
Member

hkaiser commented Sep 28, 2013

This seems to have been fixed with the recent commits

@hkaiser hkaiser closed this as completed Sep 28, 2013
@sameershende sameershende mentioned this issue Oct 26, 2013
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hkaiser hkaiser

Labels
category: threadmanager difficulty: medium project: XPRESS type: defect type: race condition
Projects
None yet
Milestone
0.9.7
Development

No branches or pull requests
2 participants
@brycelelbach @hkaiser