Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when querying performance counters #1198

Closed
parsa opened this issue Jul 18, 2014 · 4 comments
Closed

Error when querying performance counters #1198

parsa opened this issue Jul 18, 2014 · 4 comments
Assignees
@hkaiser
Labels
category: performance counters type: defect
Milestone
0.9.9

Comments

@parsa
Copy link
Contributor

parsa commented Jul 18, 2014

When you try to run mini_ghost or 1d_stencil_* by themselves they run just fine, but when you query a performance counter you'll get a segmentation fault error. If compiled in debug mode, you'll get this:

assertion 'NULL != runtime::runtime_.get()' failed: HPX(assertion_failure)
Sample command:
./mini_ghost --hpx:print-counter=/agas{locality#0/total}/count/resolve_locality

GDB Trace:

#0  0x00002aaab02761d5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00002aaab0279388 in __GI_abort () at abort.c:90
#2  0x00002aaaacecc466 in hpx::detail::assertion_failed_msg (msg=0x2aaaad8341c0 "NULL != runtime::runtime_.get()", expr=0x2aaaad8341c0 "NULL != runtime::runtime_.get()", 
    function=0x2aaaad835820 <hpx::get_runtime()::__PRETTY_FUNCTION__> "hpx::runtime& hpx::get_runtime()", file=0x2aaaad833840 "/home/pamini/at1/repo/src/runtime.cpp", line=756)
    at /home/pamini/at1/repo/src/exception.cpp:349
#3  0x00002aaaacecc1de in hpx::detail::assertion_failed (expr=0x2aaaad8341c0 "NULL != runtime::runtime_.get()", 
    function=0x2aaaad835820 <hpx::get_runtime()::__PRETTY_FUNCTION__> "hpx::runtime& hpx::get_runtime()", file=0x2aaaad833840 "/home/pamini/at1/repo/src/runtime.cpp", line=756)
    at /home/pamini/at1/repo/src/exception.cpp:309
#4  0x0000555555e49298 in hpx::assertion_failed (expr=0x2aaaad8341c0 "NULL != runtime::runtime_.get()", 
    function=0x2aaaad835820 <hpx::get_runtime()::__PRETTY_FUNCTION__> "hpx::runtime& hpx::get_runtime()", file=0x2aaaad833840 "/home/pamini/at1/repo/src/runtime.cpp", line=756)
    at /home/pamini/at1/repo/hpx/exception.hpp:1503
#5  0x00002aaaacf95c36 in hpx::get_runtime () at /home/pamini/at1/repo/src/runtime.cpp:756
#6  0x00002aaaacf9750d in hpx::naming::get_agas_client () at /home/pamini/at1/repo/src/runtime.cpp:1112
#7  0x00002aaaad45c493 in hpx::naming::detail::decrement_refcnt (p=0x2aaab78c1e80) at /home/pamini/at1/repo/src/runtime/naming/name.cpp:125
#8  0x00002aaaad45c845 in hpx::naming::detail::gid_managed_deleter (p=0x2aaab78c1e80) at /home/pamini/at1/repo/src/runtime/naming/name.cpp:183
#9  0x00002aaaad45d2d7 in hpx::naming::detail::intrusive_ptr_release (p=0x2aaab78c1e80) at /home/pamini/at1/repo/src/runtime/naming/name.cpp:413
#10 0x0000555555e51ed1 in boost::intrusive_ptr<hpx::naming::detail::id_type_impl>::~intrusive_ptr (this=0x2aaab78c7900, __in_chrg=<optimized out>)
    at /opt/boost/1.55.0-release/boost/smart_ptr/intrusive_ptr.hpp:97
#11 0x0000555555e455e0 in hpx::naming::id_type::~id_type (this=0x2aaab78c7900, __in_chrg=<optimized out>) at /home/pamini/at1/repo/hpx/runtime/naming/id_type.hpp:33
#12 0x0000555555e9188a in hpx::util::detail::value_or_error<hpx::naming::id_type>::destruct_value (this=0x2aaab78c7900)
    at /home/pamini/at1/repo/hpx/util/detail/value_or_error.hpp:294
#13 0x0000555555e87b57 in hpx::util::detail::value_or_error<hpx::naming::id_type>::destruct (this=0x2aaab78c7900) at /home/pamini/at1/repo/hpx/util/detail/value_or_error.hpp:312
#14 0x0000555555e7e894 in hpx::util::detail::value_or_error<hpx::naming::id_type>::~value_or_error (this=0x2aaab78c7900, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/util/detail/value_or_error.hpp:100
#15 0x0000555555ea05ee in hpx::lcos::detail::future_data<hpx::naming::id_type>::~future_data (this=0x2aaab78c78e8, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/lcos/detail/future_data.hpp:134
#16 0x0000555555eabfda in hpx::lcos::detail::promise<hpx::naming::id_type, hpx::naming::gid_type>::~promise (this=0x2aaab78c78d0, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/lcos/promise.hpp:218
#17 0x0000555555eac01e in hpx::lcos::detail::promise<hpx::naming::id_type, hpx::naming::gid_type>::~promise (this=0x2aaab78c78d0, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/lcos/promise.hpp:218
#18 0x0000555555e4b51d in hpx::lcos::detail::intrusive_ptr_release (p=0x2aaab78c78e8) at /home/pamini/at1/repo/hpx/lcos/detail/future_data.hpp:80
#19 0x0000555555e68eaf in boost::intrusive_ptr<hpx::lcos::detail::future_data<hpx::naming::id_type> >::~intrusive_ptr (
    this=0x55555643ea78 <hpx::util::static_<hpx::lcos::barrier, hpx::lcos::barrier>::get_reference()::t>, __in_chrg=<optimized out>)
    at /opt/boost/1.55.0-release/boost/smart_ptr/intrusive_ptr.hpp:97
#20 0x0000555555e580ec in hpx::lcos::detail::future_base<hpx::lcos::shared_future<hpx::naming::id_type>, hpx::naming::id_type>::~future_base (
    this=0x55555643ea78 <hpx::util::static_<hpx::lcos::barrier, hpx::lcos::barrier>::get_reference()::t>, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/lcos/future.hpp:441
#21 0x0000555555e58106 in hpx::lcos::shared_future<hpx::naming::id_type>::~shared_future (
    this=0x55555643ea78 <hpx::util::static_<hpx::lcos::barrier, hpx::lcos::barrier>::get_reference()::t>, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/lcos/future.hpp:1054
#22 0x0000555555e4ddc2 in hpx::components::client_base<hpx::lcos::barrier, hpx::lcos::stubs::barrier>::~client_base (
    this=0x55555643ea78 <hpx::util::static_<hpx::lcos::barrier, hpx::lcos::barrier>::get_reference()::t>, __in_chrg=<optimized out>)
    at /home/pamini/at1/repo/hpx/runtime/components/client_base.hpp:127
#23 0x0000555555e4ff68 in hpx::lcos::barrier::~barrier (this=0x55555643ea78 <hpx::util::static_<hpx::lcos::barrier, hpx::lcos::barrier>::get_reference()::t>, 
    __in_chrg=<optimized out>) at /home/pamini/at1/repo/hpx/lcos/barrier.hpp:16
#24 0x00002aaab027aa91 in __run_exit_handlers (status=0, listp=0x2aaab05e75c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#25 0x00002aaab027ab15 in __GI_exit (status=<optimized out>) at exit.c:99
@sithhell
Copy link
Member

Just saw it... That error means that the runtime isn't up... Does it happen
with other applications as well?

@sithhell
Copy link
Member

Can you please attach the actual error message?

@parsa
Copy link
Contributor Author

parsa commented Jul 18, 2014

I haven't tried it with many examples, but doesn't happen with the quickstart examples (jacobi, fibonacci).

This is the message it yields when you run mini_ghost, since you've asked. I've included the gdb trace above as well (ran with the same command line as below):

./mini_ghost --hpx:threads 4 --hpx:print-counter=/agas{locality#0/total}/count/resolve_locality --hpx:print-counter=/agas{locality#*/total}/count/bind_gid --hpx:print-counter=/agas{locality#*/total}/count/bind --hpx:print-counter=/agas{locality#0/total}/time/resolved_localities
mini ghost started up in 0.171735 seconds.

======================================================================

        Mantevo miniapp MiniGhost experiment
        HPX port

======================================================================

Computation: 5 pt difference stencil on a 2D grid (STENCIL_2D5PT)

        Global Grid Dimension: 100, 100, 100
        Local Grid Dimension : 100, 100, 100

Number of variables: 5

Error reported every 100 time steps. Tolerance is 1e-05
Number of variables reduced each time step: 0; requested 0%

        Time Steps: 100
        Task grid : 1, 1, 1

HPX version, weak scaling
1 process executing

Program execution date 2014-Jul-18 13:07:09

Total runtime: 15.8012
4 Total_time: 14.9179
finalizing ...
Runtime is not available, reporting error locally. 
{env}: 33 entries:
  CPATH=/opt/intel/composer_xe_2013_sp1.2.144/mkl/include:/opt/intel/composer_xe_2013_sp1.2.144/ipp/include:/opt/intel/composer_xe_2013_sp1.2.144/mkl/include:/opt/intel/composer_xe_2013_sp1.2.144/tbb/include
  EDITOR=vim
  GDBSERVER_MIC=/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/target/mic/bin/gdbserver
  GDB_CROSS=/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64_mic/py27/bin/gdb-mic
  HOME=/home/pamini
  IDB_HOME=/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64
  INCLUDE=/opt/intel/composer_xe_2013_sp1.2.144/mkl/include:/opt/intel/composer_xe_2013_sp1.2.144/mkl/include
  INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013_sp1.2.144/licenses:/opt/intel/licenses:/home/pamini/intel/licenses
  IPPROOT=/opt/intel/composer_xe_2013_sp1.2.144/ipp
  LANG=en_US.UTF-8
  LD_LIBRARY_PATH=/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4
  LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4
  LOGNAME=pamini
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MAIL=/var/mail/pamini
  MANPATH=/opt/intel/composer_xe_2013_sp1.2.144/man/en_US:/opt/intel/composer_xe_2013_sp1.2.144/man/en_US:/opt/intel/composer_xe_2013_sp1.2.144/man/en_US:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64_mic/py27/share/man:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64/py27/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man::/opt/intel/vtune_amplifier_xe_2013/man
  MIC_LD_LIBRARY_PATH=/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
  MIC_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
  MKLROOT=/opt/intel/composer_xe_2013_sp1.2.144/mkl
  NLSPATH=/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64_mic/py27/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64/py27/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.2.144/debugger/intel64/locale/%l_%t/%N
  OLDPWD=/home/pamini/at1
  PATH=/opt/intel/vtune_amplifier_xe_2013/bin64:/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/bin/intel64:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64_mic/py27/bin:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64/py27/bin:/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64:/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64_mic:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gui/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  PWD=/home/pamini/at1/build/bin
  SHELL=/bin/bash
  SHLVL=1
  SSH_CLIENT=96.125.89.60 50503 22
  SSH_CONNECTION=96.125.89.60 50503 10.1.1.11 22
  SSH_TTY=/dev/pts/13
  TBBROOT=/opt/intel/composer_xe_2013_sp1.2.144/tbb
  TERM=xterm-256color
  USER=pamini
  VTUNE_AMPLIFIER_XE_2013_DIR=/opt/intel/vtune_amplifier_xe_2013
  _=./mini_ghost
{locality-id}: 4294967295
{process-id}: 32305
{function}: hpx::runtime& hpx::get_runtime()
{file}: /home/pamini/at1/repo/src/runtime.cpp
{line}: 756
{os-thread}: <unknown>
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_IPC=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=ON
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_LIMIT=5
  HPX_ACTION_ARGUMENT_LIMIT=5
  HPX_COMPONENT_CREATE_ARGUMENT_LIMIT=5
  HPX_FUNCTION_ARGUMENT_LIMIT=8
  HPX_LOCK_LIMIT=5
  HPX_TUPLE_LIMIT=8
  HPX_WAIT_ARGUMENT_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX=/home/pamini/at1/build:/usr/local
{version}: V0.9.9-trunk (AGAS: V3.0), Git: ee1d748efbb3be3164359af81a2da12e3b9a6cb8
{boost}: V1.55.0
{build-type}: debug
{date}: Jul 18 2014 11:42:38
{platform}: linux
{compiler}: GNU C++ version 4.9.0
{stdlib}: GNU libstdc++ version 20140704
{what}: assertion 'NULL != runtime::runtime_.get()' failed: HPX(assertion_failure)
Aborted

@hkaiser hkaiser added this to the 0.9.9 milestone Jul 18, 2014
@hkaiser hkaiser self-assigned this Jul 18, 2014
@hkaiser
Copy link
Member

hkaiser commented Jul 18, 2014

Working on it...

@parsa parsa mentioned this issue Jul 22, 2014
Closed
hkaiser added a commit that referenced this issue Jul 28, 2014
@hkaiser
Fixed #1198: Error when querying performance counters, improved error…
54c39f7 
… reporting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hkaiser hkaiser

Labels
category: performance counters type: defect
Projects
None yet
Milestone
0.9.9
Development

No branches or pull requests
3 participants
@sithhell @parsa @hkaiser