Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while using counters in HPX #585

Closed
sameershende opened this issue Oct 25, 2012 · 7 comments
Closed

Error while using counters in HPX #585

sameershende opened this issue Oct 25, 2012 · 7 comments

Comments

@sameershende
Copy link

Hi,
I am using the HPX (git head) and I can run the code properly on our cluster without counters. When I use counters, I get an error and a stack trace. Matt Anderson does not see this problem with the same binary. What could be going wrong? I see:

qsub -I -V -q gpu -l nodes=2 -d /home3/sameer

cd /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/bin

setenv BOOST_ROOT /ibrix/home0/andersmw_tau/boost_1_51_0

[sameer@cn70 bin]$ pbsdsh `pwd`/hello_world --hpx:nodes=`cat $PBS_NODEFILE` 
hello world from OS-thread 0 on locality 0
hello world from OS-thread 0 on locality 1

[sameer@cn70 bin]$ pbsdsh `pwd`/hello_world --hpx:nodes=`cat $PBS_NODEFILE` \
    --hpx:print-counter=/messages{locality#0/total}/count/sent

{stack-trace}: 10 frames:
0x7f5e6597568c : hpx::detail::backtrace() + 0x5c in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e659983f6 : boost::exception_ptr hpx::detail::get_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x46 in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e659986ba : void hpx::detail::throw_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x1a in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65cf09db : hpx::util::query_counters::find_counters() + 0x33b in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65cf0d40 : hpx::util::query_counters::start() + 0x10 in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65b1359b : hpx::components::server::runtime_support::call_startup_functions(bool) + 0x5b in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65b29b6a : ??? + 0x7f5e65b29b6a in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65c1b1e3 : ??? + 0x7f5e65c1b1e3 in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
0x7f5e65c09519 : ??? + 0x7f5e65c09519 in /usr/local/packages/HPX/apps/hpx_matt_boost_1510.un/lib/hpx/libhpx.so.0
{env}: 87 entries:
BOOST_ROOT=/usr/local/packages/HPX/apps/boost_1_51_0/
CMAKE=/usr/local/packages/cmake/2.8.6
CPLUS_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
CTBIN=/opt/clustertest/bin
CTLOGS=/opt/clustertest/logs
CTTEST=/opt/clustertest/logs/under_test
C_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
DISPLAY=localhost:40.0
DYNINSTAPI_RT_LIB=/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib/libdyninstAPI_RT.so.7
DYNINST_ROOT=/usr/local/packages/dyninstAPI-7.0.2
ENVIRONMENT=BATCH
GCC=/usr/local/packages/gcc/4.6.3
GMP=/usr/local/packages/gmp/5.0.5
GROUP=cas_cis
G_BROKEN_FILENAMES=1
HDF5_CPP=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_FORTRAN_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HOME=/home3/sameer
HOST=hn1
HOSTNAME=hn1
HOSTTYPE=x86_64-linux
INFOPATH=/usr/local/packages/gmp/5.0.5/share/info:/usr/local/packages/mpfr/3.1.0/share/info:/usr/local/packages/mpc/0.9-2/share/info:/usr/local/packages/gcc/4.6.3/info
LANG=en_US.UTF-8
LD_DYNAMIC_WEAK=1
LD_LIBRARY_PATH=/ibrix/home3/sameer/apps/instr/mm/mylib:/usr/local/packages/gmp/5.0.5/lib:/usr/local/packages/mpfr/3.1.0/lib:/usr/local/packages/mpc/0.9-2/lib:/usr/local/packages/gcc/4.6.3/lib64:/usr/local/packages/gcc/4.6.3/lib:/home3/sameer/tau2//x86_64/lib:/usr/local/packages/libdwarf-20111030/lib:/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOADEDMODULES=cmake/2.8.6:gcc/4.6.3
LOGNAME=sameer
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.tbz=01;31:.tbz2=01;31:.bz=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:.xspf=01;36:
MACHTYPE=x86_64
MAIL=/var/spool/mail/sameer
MANPATH=/usr/local/packages/gcc/4.6.3/share/man:/usr/local/packages/cmake/2.8.6/man:/opt/torque/man:/usr/local/share/man:/usr/share/man
MODULEPATH=/usr/local/packages/Modules/modulefiles
MODULESHOME=/usr/share/Modules
MPC=/usr/local/packages/mpc/0.9-2
MPFR=/usr/local/packages/mpfr/3.1.0
OMP_NUM_THREADS=4
OSTYPE=linux
PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
PBS_ENVIRONMENT=PBS_BATCH
PBS_GPUFILE=/opt/torque/aux//186478.hn1gpu
PBS_JOBCOOKIE=1A9F65C8ADDF0ADE59464FDC26D8AF06
PBS_JOBID=186478.hn1
PBS_JOBNAME=STDIN
PBS_MOMPORT=15003
PBS_NODEFILE=/opt/torque/aux//186478.hn1
PBS_NODENUM=0
PBS_NUM_NODES=2
PBS_NUM_PPN=1
PBS_O_HOME=/home3/sameer
PBS_O_HOST=hn1
PBS_O_INITDIR=/home3/sameer

PBS_O_LANG=en_US.UTF-8
PBS_O_LOGNAME=sameer
PBS_O_MAIL=/var/spool/mail/sameer
PBS_O_PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
PBS_O_QUEUE=gpu
PBS_O_SHELL=/bin/tcsh
PBS_O_WORKDIR=/home3/sameer
PBS_QUEUE=gpu
PBS_SERVER=hn1
PBS_TASKNUM=10
PBS_VERSION=TORQUE-2.5.12
PBS_VNODENUM=0
PBS_WALLTIME=86400
PKG=/usr/local/packages
PLATFORM=x86_64-unknown-linux2.4
PWD=/home3/sameer
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
QTLIB=/usr/lib64/qt-3.3/lib
REMOTEHOST=dhcp-207.nic.uoregon.edu
SH=/usr/local/packages/tar/openshmem/examples
SHELL=/bin/tcsh
SHLVL=1
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_AUTH_SOCK=/tmp/ssh-AjftL17708/agent.17708
SSH_CLIENT=128.223.202.207 54301 22
SSH_CONNECTION=128.223.202.207 54301 128.223.224.254 22
SSH_TTY=/dev/pts/3
T=/home3/sameer/tau2/
TAU_MAKEFILE=/home3/sameer/tau2//include/Makefile
TERM=xterm
USER=sameer
VENDOR=unknown
LMFILES=/usr/local/packages/Modules/modulefiles/cmake/2.8.6:/usr/local/packages/Modules/modulefiles/gcc/4.6.3
{what}: this future has not been initialized: HPX(broken_promise)
{locality-id}: 0
{hostname}: 10.0.1.70:7910
{process-id}: 23850
{function}: future::get
{file}: /home3/sameer/apps/hpx/hpx/lcos/future.hpp
{line}: 134
{os-thread}: 0
{thread-id}: 00007f5e673810b0
{thread-description}:
{version}: V0.9.5-trunk (AGAS: V2.1), Git:
{boost}: V1.51.0
{build-type}: release
{date}: Oct 24 2012 17:20:12
{platform}: linux
{compiler}: GNU C++ version 4.6.3
{stdlib}: GNU libstdc++ version 20120301

pbsdsh: task 0 exit status 262

I should add that Matt Anderson does not see these errors on our system when we both have the same set of modules and use the same binary. What else could affect this execution?
Thanks,

  • Sameer
@ghost ghost assigned hkaiser Oct 25, 2012
@hkaiser
Copy link
Member

hkaiser commented Oct 25, 2012

It's very difficult for me to guess what happens. Thus, I did add more error reporting around this particular problem which should give us more insights of what actually goes wrong for you (see c05ffa6). May I ask you to check out my changes, recompile, and report back with whatever error messages you might see?

@hkaiser
Copy link
Member

hkaiser commented Oct 28, 2012

One of our recent commits (see 5a6f43a) might have fixed the issue which was causing your trouble. I'd be interested in seeing whether things do work for you now.

@sameershende
Copy link
Author

Hi Hartmut,
Unfortunately, even after an svn update, it didn't work. I made sure that my stacksize was unlimited, but it made no difference. I get:
gdb ./hello_world
(gdb) run -t 1 --hpx:print-counter=/parcels{locality#0/total}/count/sent
Starting program: /ibrix/packages/HPX/apps/hpx_gcc463.un/bin/hello_world -t 1 --hpx:print-counter=/parcels{locality#0/total}/count/sent
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaab4578700 (LWP 13912)]
[New Thread 0x2aaab4779700 (LWP 13913)]
[New Thread 0x2aaac0200700 (LWP 13914)]
[New Thread 0x2aaac0401700 (LWP 13915)]
[New Thread 0x2aaac0602700 (LWP 13916)]
[New Thread 0x2aaacc200700 (LWP 13917)]
[New Thread 0x2aaad8200700 (LWP 13918)]
[New Thread 0x2aaad8602700 (LWP 13919)]

{stack-trace}: 14 frames:
0x2aaaab47d14c : hpx::detail::backtrace() + 0x5c in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab49fe76 : boost::exception_ptr hpx::detail::get_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x46 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab4a013a : void hpx::detail::throw_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x1a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab7bf712 : hpx::performance_counters::get_counter_type_path_elements(std::string const&, hpx::performance_counters::counter_type_path_elements&, hpx::error_code&) + 0x3c2 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab7bf7ca : hpx::performance_counters::get_counter_type_name(std::string const&, std::string&, hpx::error_code&) + 0x3a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab74d8ea : ??? + 0x2aaaab74d8ea in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab7b8bf0 : hpx::performance_counters::discover_counter_type(std::string const&, hpx::util::function_nonser<bool (hpx::performance_counters::counter_info const&, hpx::error_code&)> const&, hpx::performance_counters::discover_counters_mode, hpx::error_code&) + 0x90 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab807a08 : hpx::util::query_counters::find_counters() + 0x248 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab807c30 : hpx::util::query_counters::start() + 0x10 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab61f9db : hpx::components::server::runtime_support::call_startup_functions(bool) + 0x5b in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab63674a : ??? + 0x2aaaab63674a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab72aa23 : ??? + 0x2aaaab72aa23 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2aaaab719299 : ??? + 0x2aaaab719299 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
{env}: 91 entries:
BOOST_ROOT=/usr/local/packages/HPX/apps/boost_1_51_0/
CMAKE=/usr/local/packages/cmake/2.8.6
COLUMNS=80
CPLUS_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
CTBIN=/opt/clustertest/bin
CTLOGS=/opt/clustertest/logs
CTTEST=/opt/clustertest/logs/under_test
C_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
...
{date}: Oct 29 2012 15:57:52
{platform}: linux
{compiler}: GNU C++ version 4.6.3
{stdlib}: GNU libstdc++ version 20120301

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x2aaad8200700 (LWP 13918)]
0x00002aaaae0738a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.5.x86_64
(gdb) where
#0 0x00002aaaae0738a5 in raise () from /lib64/libc.so.6
#1 0x00002aaaae075085 in abort () from /lib64/libc.so.6
#2 0x00002aaaab47e136 in hpx::detail::report_exception_and_terminate(boost::exception_ptr const&) ()
from /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
#3 0x00002aaaab4c1030 in hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler, hpx::threads::policies::callback_notifier>::report_error(unsigned long, boost::exception_ptr const&) ()
from /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
#4 0x00002aaaab731008 in hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler, hpx::threads::policies::callback_notifier>::tfunc(unsigned long) ()
from /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
#5 0x00002aaaacdfda5d in thread_proxy ()
from /usr/local/packages/HPX/apps/boost_1_51_0/stage/lib/libboost_thread.so.1.51.0
#6 0x00002aaaade2b851 in start_thread () from /lib64/libpthread.so.0
#7 0x00002aaaae12911d in clone () from /lib64/libc.so.6
(gdb)
(gdb)
(gdb) quit
A debugging session is active.

    Inferior 1 [process 13894] will be killed.

Quit anyway? (y or n) y

Here is another script output. Perhaps we need to get you an account on our machine or identify another system where HPX is being used where we can all get accounts. 
Thanks,
- Sameer

On Oct 28, 2012, at 3:30 PM, Hartmut Kaiser notifications@github.com wrote:

One of our recent commits (see 5a6f43a) might have fixed the issue which was causing your trouble. I'd be interested in seeing whether things do work for you now.


Reply to this email directly or view it on GitHub.

Script started on Mon 29 Oct 2012 04:37:08 PM PDT
limit
�]0;sameer@cn161:~/hb�[sameer@cn161 /hb]$ limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize 0 kbytes
memoryuse unlimited
vmemoryuse unlimited
descriptors 32768
memorylocked unlimited
maxproc 1024
�]0;sameer@cn161:
/hb�[sameer@cn161 /hb]$ ls
�[0m�[01;32m1d_wave_equation�[0m �[01;32mfractals�[0m �[01;32mhpx_run_tests.py�[0m
�[01;32mcomponent_ctors�[0m �[01;32mgtcx_client�[0m �[01;32minterest_calculator�[0m
�[01;32mcomponent_in_executable�[0m �[01;32mhello_world�[0m �[01;32mnon_atomic_rma�[0m
�[01;32mcomponent_inheritance�[0m �[01;36mhpx_environment�[0m �[01;32mpingpong�[0m
�[01;32merror_handling�[0m �[01;32mhpx_environment.py�[0m �[01;32mquicksort�[0m
�[01;32mevent_synchronization�[0m �[01;36mhpx_invoke�[0m �[01;32msimplest_hello_world�[0m
�[01;32mfactorial�[0m �[01;32mhpx_invoke.py�[0m �[01;32mtimed_wake�[0m
�[01;32mfibonacci�[0m �[01;36mhpx_optsweep�[0m typescript
�[01;32mfibonacci_futures�[0m �[01;32mhpx_optsweep.py�[0m �[01;32muse_main_thread�[0m
�[01;32mfibonacci_one�[0m �[01;36mhpx_run_tests�[0m �[01;32mwait_composition�[0m
�[m�]0;sameer@cn161:
/hb�[sameer@cn161 ~/hb]$ pbsdsh -v -u pwd/gtcx_client --hpx:nodes=cat $PBS_NODEF �ILE --hpx:debug-clp --hpx:print-counter=/parcels{locality#0/total}/count/sent
pbsdsh: rescinfo from 0: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 1: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 2: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 3: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 4: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 5: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 6: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 7: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 8: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 9: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 10: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 11: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 12: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 13: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 14: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 15: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 16: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 17: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 18: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 19: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 20: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 21: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 22: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 23: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: spawned task 0
pbsdsh: spawned task 1
pbsdsh: spawn event returned: 0 (2 spawns and 0 obits outstanding)
pbsdsh: sending obit for task 2
pbsdsh: spawn event returned: 1 (1 spawns and 1 obits outstanding)
pbsdsh: sending obit for task 3
pbsdsh: obit event returned: 0 (0 spawns and 2 obits outstanding)
pbsdsh: task 0 exit status 262

^C^C^C
^Z
Suspended
�]0;sameer@cn161:~/hb�[sameer@cn161 /hb]$ ps -u samer��[Ker
PID TTY TIME CMD
11134 pts/0 00:00:00 tcsh
11135 pts/0 00:00:00 pbs_demux
11150 pts/0 00:00:00 script
11152 pts/0 00:00:00 script
11153 pts/1 00:00:00 tcsh
11206 pts/1 00:00:00 pbsdsh
11292 pts/1 00:00:00 ps
�]0;sameer@cn161:
/hb�[sameer@cn161 /hb]$ kill -9 11135 1120t6��[K��[K6
�]0;sameer@cn161:
/hb�[sameer@cn161 /hb]$
[1] Killed pbsdsh -v -u pwd/gtcx_client --hpx:nodes=cat $PBS_NODEFILE --hpx:debug-clp --hpx:print-counter=/parcels{locality#0/total}/count/sent
�]0;sameer@cn161:
/hb�[sameer@cn161 ~/hb]$ pbsdsh -v -u pwd/gtcx_client --hpx:nodes=cat $PBS_NODEF �ILE --hpx:debug-clp
pbsdsh: rescinfo from 0: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 12: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 13: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 14: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 15: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 16: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 17: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 18: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 19: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 20: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 21: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 1: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 22: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 23: Linux cn94 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 2: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 3: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 4: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 5: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 6: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 7: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 8: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 9: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 10: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: rescinfo from 11: Linux cn161 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64:nodes=2:ppn=12,walltime=24:00:00
pbsdsh: spawned task 0
pbsdsh: spawned task 1
pbsdsh: spawn event returned: 0 (2 spawns and 0 obits outstanding)
pbsdsh: sending obit for task 4
pbsdsh: spawn event returned: 1 (1 spawns and 1 obits outstanding)
pbsdsh: sending obit for task 5
pbsdsh: obit event returned: 1 (0 spawns and 2 obits outstanding)
pbsdsh: task 1 exit status 255

^C^C^C
^Z
Suspended
�]0;sameer@cn161:~/hb�[sameer@cn161 /hb]$ env
PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
LANG=en_US.UTF-8
PBS_O_QUEUE=generic
PBS_O_HOST=hn1
PBS_O_HOME=/home3/sameer
PBS_O_LANG=en_US.UTF-8
PBS_O_LOGNAME=sameer
PBS_O_PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
PBS_O_MAIL=/var/spool/mail/sameer
PBS_O_SHELL=/bin/tcsh
PBS_SERVER=hn1
PBS_O_INITDIR=/home3/sameer
PBS_O_WORKDIR=/home3/sameer
USER=sameer
LOGNAME=sameer
HOME=/home3/sameer
MAIL=/var/spool/mail/sameer
SHELL=/bin/tcsh
SSH_CLIENT=128.223.202.207 59765 22
SSH_CONNECTION=128.223.202.207 59765 128.223.224.254 22
SSH_TTY=/dev/pts/19
TERM=xterm
DISPLAY=localhost:21.0
SSH_AUTH_SOCK=/tmp/ssh-zcclo18055/agent.18055
HOSTTYPE=x86_64-linux
VENDOR=unknown
OSTYPE=linux
MACHTYPE=x86_64
SHLVL=2
PWD=/home3/sameer/hb
GROUP=cas_cis
HOST=cn161
REMOTEHOST=dhcp-207.nic.uoregon.edu
HOSTNAME=cn161
CTBIN=/opt/clustertest/bin
CTLOGS=/opt/clustertest/logs
CTTEST=/opt/clustertest/logs/under_test
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.tbz=01;31:.tbz2=01;31:.bz=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:.xspf=01;36:
G_BROKEN_FILENAMES=1
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
LESSOPEN=|/usr/bin/lesspipe.sh %s
LD_LIBRARY_PATH=/ibrix/home3/sameer/apps/instr/mm/mylib:/usr/local/packages/gmp/5.0.5/lib:/usr/local/packages/mpfr/3.1.0/lib:/usr/local/packages/mpc/0.9-2/lib:/usr/local/packages/gcc/4.6.3/lib64:/usr/local/packages/gcc/4.6.3/lib:/home3/sameer/tau2//x86_64/lib:/usr/local/packages/libdwarf-20111030/lib:/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib
MANPATH=/usr/local/packages/gcc/4.6.3/share/man:/usr/local/packages/cmake/2.8.6/man:/opt/torque/man:/usr/local/share/man:/usr/share/man
MODULESHOME=/usr/share/Modules
MODULEPATH=/usr/local/packages/Modules/modulefiles
LOADEDMODULES=cmake/2.8.6:gcc/4.6.3
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
QTLIB=/usr/lib64/qt-3.3/lib
T=/home3/sameer/tau2/
SH=/usr/local/packages/tar/openshmem/examples
TAU_MAKEFILE=/home3/sameer/tau2//include/Makefile
PLATFORM=x86_64-unknown-linux2.4
PKG=/usr/local/packages
LD_DYNAMIC_WEAK=1
DYNINST_ROOT=/usr/local/packages/dyninstAPI-7.0.2
DYNINSTAPI_RT_LIB=/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib/libdyninstAPI_RT.so.7
CMAKE=/usr/local/packages/cmake/2.8.6
LMFILES=/usr/local/packages/Modules/modulefiles/cmake/2.8.6:/usr/local/packages/Modules/modulefiles/gcc/4.6.3
CPLUS_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
C_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
INFOPATH=/usr/local/packages/gmp/5.0.5/share/info:/usr/local/packages/mpfr/3.1.0/share/info:/usr/local/packages/mpc/0.9-2/share/info:/usr/local/packages/gcc/4.6.3/info
BOOST_ROOT=/usr/local/packages/HPX/apps/boost_1_51_0/
HDF5_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_CPP=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_FORTRAN_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
OMP_NUM_THREADS=4
PBS_JOBNAME=STDIN
PBS_JOBID=187635.hn1
PBS_QUEUE=generic
PBS_JOBCOOKIE=9F8B341AA507F8A8C9BD9D9D7CD91056
PBS_NODENUM=0
PBS_TASKNUM=1
PBS_MOMPORT=15003
PBS_NODEFILE=/opt/torque/aux//187635.hn1
PBS_GPUFILE=/opt/torque/aux//187635.hn1gpu
PBS_WALLTIME=86400
PBS_NUM_NODES=2
PBS_NUM_PPN=12
PBS_VERSION=TORQUE-2.5.12
PBS_VNODENUM=0
PBS_NP=24
PBS_ENVIRONMENT=PBS_INTERACTIVE
KDEDIRS=/usr
KDE_IS_PRELINKED=1
GCC=/usr/local/packages/gcc/4.6.3
GMP=/usr/local/packages/gmp/5.0.5
MPC=/usr/local/packages/mpc/0.9-2
MPFR=/usr/local/packages/mpfr/3.1.0
�]0;sameer@cn161:
/hb�[sameer@cn161 /hb]$ ^D��
There are suspended jobs.
�]0;sameer@cn161:
/hb�[sameer@cn161 ~/hb]$ exit
exit

Script done on Mon 29 Oct 2012 04:39:13 PM PDT

@hkaiser
Copy link
Member

hkaiser commented Oct 30, 2012

Sameer, unfortunately you cut out the important parts of the error output. Please run again (outside of gdb) and send us the whole output you get.

Additionally, may I ask you to add the command line options --hpx:debug-clp and --hpx:ini=hpx.logging.level=4. This will create a file in the current working directory named hpx..log (where is the PID of the job you ran). Please attach this file as well.

Could you do an independent run while adding --hpx:list-counters, please and send the list of performance counters it prints as well.

Thanks!

@sameershende
Copy link
Author

Hi Hartmut,
I have attached the error logs.
Thanks,
- Sameer

On Oct 29, 2012, at 5:31 PM, Hartmut Kaiser notifications@github.com wrote:

Sameer, unfortunately you cut out the important parts of the error output. Please run again (outside of gdb) and send us the whole output you get.

Additionally, may I ask you to add the command line options --hpx:debug-clp and --hpx:ini=hpx.logging.level=4. This will create a file in the current working directory named hpx..log (where is the PID of the job you ran). Please attach this file as well.

Could you do an independent run while adding --hpx:list-counters, please and send the list of performance counters it prints as well.

Thanks!


Reply to this email directly or view it on GitHub.

Script started on Mon 29 Oct 2012 05:32:24 PM PDT
�]0;sameer@cn167:bin�[sameer@cn167 bin]$ ./hello_world --hpx:debug-clp --hpx:ini=hpx.logging.level=4 --hpx:print-counter=/parcels{locality#0/total}/count/sent
agas host_name: 127.0.0.1
host_name: 127.0.0.1
retrieve_number_of_threads: 1
retrieve_number_of_localities: 1
retrieve_node_number (PBS_NODENUM/SLURM_PROCID): 0
resolved: '127.0.0.1' to: 127.0.0.1
resolved: '127.0.0.1' to: 127.0.0.1

Configuration before runtime start:

hpx.locality=0
hpx.logging.level=4
hpx.parcel.address=127.0.0.1
hpx.parcel.port=7910
hpx.agas.address=127.0.0.1
hpx.agas.port=7910
hpx.agas.service_mode=bootstrap
hpx.program_name=./hello_world
hpx.cmd_line=./hello_world -t 1 --hpx:debug-clp --hpx:ini=hpx.logging.level=4 --hpx:print-counter=/parcelslocality#0/total/count/sent
hpx.reconstructed_cmd_line=./hello_world --hpx:config --hpx:debug-clp --hpx:ini=hpx.logging.level=4 --hpx:print-counter=/parcelslocality#0/total/count/sent --hpx:threads=1
hpx.os_threads=1
hpx.localities=1

hpx.runtime_mode=console

(T00000000/----------------.--/----------------) P--------/----------------.-- 17:33.57.680 [0000000000000001] [ERR] tfunc(0): caught hpx::exception: invalid counter name format: HPX(bad_parameter), aborted thread execution

{stack-trace}: 14 frames:
0x2b7f1d7f814c : hpx::detail::backtrace() + 0x5c in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1d81ae76 : boost::exception_ptr hpx::detail::get_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x46 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1d81b13a : void hpx::detail::throw_exceptionhpx::exception(hpx::exception const&, std::string const&, std::string const&, long) + 0x1a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1db3a712 : hpx::performance_counters::get_counter_type_path_elements(std::string const&, hpx::performance_counters::counter_type_path_elements&, hpx::error_code&) + 0x3c2 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1db3a7ca : hpx::performance_counters::get_counter_type_name(std::string const&, std::string&, hpx::error_code&) + 0x3a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1dac88ea : ??? + 0x2b7f1dac88ea in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1db33bf0 : hpx::performance_counters::discover_counter_type(std::string const&, hpx::util::function_nonser<bool (hpx::performance_counters::counter_info const&, hpx::error_code&)> const&, hpx::performance_counters::discover_counters_mode, hpx::error_code&) + 0x90 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1db82a08 : hpx::util::query_counters::find_counters() + 0x248 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1db82c30 : hpx::util::query_counters::start() + 0x10 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1d99a9db : hpx::components::server::runtime_support::call_startup_functions(bool) + 0x5b in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1d9b174a : ??? + 0x2b7f1d9b174a in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1daa5a23 : ??? + 0x2b7f1daa5a23 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
0x2b7f1da94299 : ??? + 0x2b7f1da94299 in /usr/local/packages/HPX/apps/hpx_gcc463.un/lib/hpx/libhpx.so.0
{env}: 89 entries:
BOOST_ROOT=/usr/local/packages/HPX/apps/boost_1_51_0/
CMAKE=/usr/local/packages/cmake/2.8.6
CPLUS_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
CTBIN=/opt/clustertest/bin
CTLOGS=/opt/clustertest/logs
CTTEST=/opt/clustertest/logs/under_test
C_INCLUDE_PATH=/usr/local/packages/gcc/4.6.3/include
DISPLAY=localhost:18.0
DYNINSTAPI_RT_LIB=/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib/libdyninstAPI_RT.so.7
DYNINST_ROOT=/usr/local/packages/dyninstAPI-7.0.2
GCC=/usr/local/packages/gcc/4.6.3
GMP=/usr/local/packages/gmp/5.0.5
GROUP=cas_cis
G_BROKEN_FILENAMES=1
HDF5_CPP=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_FORTRAN_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HDF5_ROOT=/usr/local/packages/HPX/apps/hdf5-1.8.9_gcc-4.6.3
HOME=/home3/sameer
HOST=cn167
HOSTNAME=cn167
HOSTTYPE=x86_64-linux
INFOPATH=/usr/local/packages/gmp/5.0.5/share/info:/usr/local/packages/mpfr/3.1.0/share/info:/usr/local/packages/mpc/0.9-2/share/info:/usr/local/packages/gcc/4.6.3/info
KDEDIRS=/usr
KDE_IS_PRELINKED=1
LANG=en_US.UTF-8
LD_DYNAMIC_WEAK=1
LD_LIBRARY_PATH=/ibrix/home3/sameer/apps/instr/mm/mylib:/usr/local/packages/gmp/5.0.5/lib:/usr/local/packages/mpfr/3.1.0/lib:/usr/local/packages/mpc/0.9-2/lib:/usr/local/packages/gcc/4.6.3/lib64:/usr/local/packages/gcc/4.6.3/lib:/home3/sameer/tau2//x86_64/lib:/usr/local/packages/libdwarf-20111030/lib:/usr/local/packages/dyninstAPI-7.0.2/x86_64-unknown-linux2.4/lib
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOADEDMODULES=cmake/2.8.6:gcc/4.6.3
LOGNAME=sameer
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.tbz=01;31:.tbz2=01;31:.bz=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:.xspf=01;36:
MACHTYPE=x86_64
MAIL=/var/spool/mail/sameer
MANPATH=/usr/local/packages/gcc/4.6.3/share/man:/usr/local/packages/cmake/2.8.6/man:/opt/torque/man:/usr/local/share/man:/usr/share/man
MODULEPATH=/usr/local/packages/Modules/modulefiles
MODULESHOME=/usr/share/Modules
MPC=/usr/local/packages/mpc/0.9-2
MPFR=/usr/local/packages/mpfr/3.1.0
OMP_NUM_THREADS=4
OSTYPE=linux
PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
PBS_ENVIRONMENT=PBS_INTERACTIVE
PBS_GPUFILE=/opt/torque/aux//187648.hn1gpu
PBS_JOBCOOKIE=D38E72C82E470F735B903E8CE383F173
PBS_JOBID=187648.hn1
PBS_JOBNAME=STDIN
PBS_MOMPORT=15003
PBS_NODEFILE=/opt/torque/aux//187648.hn1
PBS_NODENUM=0
PBS_NP=24
PBS_NUM_NODES=2
PBS_NUM_PPN=12
PBS_O_HOME=/home3/sameer
PBS_O_HOST=hn1
PBS_O_INITDIR=/home3/sameer
PBS_O_LANG=en_US.UTF-8
PBS_O_LOGNAME=sameer
PBS_O_MAIL=/var/spool/mail/sameer
PBS_O_PATH=/usr/local/packages/intel/composer_xe_2011_sp1/pkg_bin/intel64:/usr/local/packages/gcc/4.6.3/bin:/usr/local/packages/cmake/2.8.6/bin:/usr/local/packages/openshmem/bin:/usr/local/packages/msmpi/jcl/Bin:/usr/local/packages/mingw64/bin:/home3/sameer/tau2//x86_64/bin:/usr/lib64/qt-3.3/bin:/opt/torque/bin:/usr/local/bin:/bin:/usr/bin:/opt/clustertest/bin
PBS_O_QUEUE=generic
PBS_O_SHELL=/bin/tcsh
PBS_O_WORKDIR=/home3/sameer
PBS_QUEUE=generic
PBS_SERVER=hn1
PBS_TASKNUM=1
PBS_VERSION=TORQUE-2.5.12
PBS_VNODENUM=0
PBS_WALLTIME=86400
PKG=/usr/local/packages
PLATFORM=x86_64-unknown-linux2.4
PWD=/usr/local/packages/HPX/apps/hpx_gcc463.un/bin
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
QTLIB=/usr/lib64/qt-3.3/lib
REMOTEHOST=dhcp-207.nic.uoregon.edu
SH=/usr/local/packages/tar/openshmem/examples
SHELL=/bin/tcsh
SHLVL=2
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_AUTH_SOCK=/tmp/ssh-sSiAj16981/agent.16981
SSH_CLIENT=128.223.202.207 61173 22
SSH_CONNECTION=128.223.202.207 61173 128.223.224.254 22
SSH_TTY=/dev/pts/9
T=/home3/sameer/tau2/
TAU_MAKEFILE=/home3/sameer/tau2//include/Makefile
TERM=xterm
USER=sameer
VENDOR=unknown
LMFILES=/usr/local/packages/Modules/modulefiles/cmake/2.8.6:/usr/local/packages/Modules/modulefiles/gcc/4.6.3
{what}: invalid counter name format: HPX(bad_parameter)
{locality-id}: 0
{hostname}: 127.0.0.1:7910
{process-id}: 5069
{function}: get_counter_type_path_elements
{file}: /home3/sameer/apps/hpx/src/performance_counters/counters.cpp
{line}: 365
{os-thread}: 0
{thread-id}: 00002b7f2079d100
{thread-description}:
{version}: V0.9.5-trunk (AGAS: V2.1), Git:
{boost}: V1.51.0
{build-type}: release
{date}: Oct 29 2012 15:57:52
{platform}: linux
{compiler}: GNU C++ version 4.6.3
{stdlib}: GNU libstdc++ version 20120301

Abort (core dumped)
�]0;sameer@cn167:bin�[sameer@cn167 bin]$ ./hello_word --hpx:list-counters�[21G./hello_wor�[1@l
List of available counter instances

(replace '*' below with the appropriate sequence number)

/agas/count/allocate
/agas/count/bind
/agas/count/bind_gid
/agas/count/bind_name
/agas/count/bind_prefix
/agas{locality#/total}/count/cache-evictions
/agas{locality#
/total}/count/cache-hits
/agas{locality#/total}/count/cache-insertions
/agas{locality#
/total}/count/cache-misses
/agas/count/change_credit_non_blocking
/agas/count/change_credit_sync
/agas/count/free
/agas/count/get_component_typename
/agas/count/iterate_names
/agas/count/iterate_types
/agas/count/localities
/agas/count/num_localities
/agas/count/num_threads
/agas/count/resolve
/agas/count/resolve_gid
/agas/count/resolve_id
/agas/count/resolve_locality
/agas/count/unbind
/agas/count/unbind_gid
/agas/count/unbind_name
/agas/time/allocate
/agas/time/bind
/agas/time/bind_gid
/agas/time/bind_name
/agas/time/bind_prefix
/agas/time/change_credit_non_blocking
/agas/time/change_credit_sync
/agas/time/free
/agas/time/get_component_typename
/agas/time/iterate_names
/agas/time/iterate_types
/agas/time/localities
/agas/time/num_localities
/agas/time/num_threads
/agas/time/resolve
/agas/time/resolve_gid
/agas/time/resolve_id
/agas/time/resolve_locality
/agas/time/unbind
/agas/time/unbind_gid
/agas/time/unbind_name
/data{locality#/total}/count/received
/data{locality#
/total}/count/sent
/data{locality#/total}/time/received
/data{locality#
/total}/time/sent
/dataflow{locality#/total}/count/constructed
/dataflow{locality#
/total}/count/destructed
/dataflow{locality#/total}/count/fired
/dataflow{locality#
/total}/count/initialized
/full_empty{locality#/total}/count/constructed
/full_empty{locality#
/total}/count/destructed
/full_empty{locality#/total}/count/fired
/full_empty{locality#
/total}/count/read_dequeued
/full_empty{locality#/total}/count/read_enqueued
/messages{locality#
/total}/count/received
/messages{locality#/total}/count/sent
/parcelqueue{locality#
/total}/length/receive
/parcelqueue{locality#/total}/length/send
/parcels{locality#
/total}/count/received
/parcels{locality#/total}/count/sent
/runtime{locality#
/total}/count/component
/runtime{locality#/total}/uptime
/serialize{locality#
/total}/count/received
/serialize{locality#/total}/count/sent
/serialize{locality#
/total}/time/received
/serialize{locality#/total}/time/sent
/statistics/average
/statistics/max
/statistics/min
/threadqueue{locality#
/total}/length
/threadqueue{locality#/worker-thread#}/length
/threads{locality#/total}/count/cumulative
/threads{locality#
/worker-thread#}/count/cumulative
/threads{locality#
/total}/count/instantaneous/active
/threads{locality#/worker-thread#}/count/instantaneous/active
/threads{locality#/total}/count/instantaneous/all
/threads{locality#
/worker-thread#}/count/instantaneous/all
/threads{locality#
/total}/count/instantaneous/pending
/threads{locality#/worker-thread#}/count/instantaneous/pending
/threads{locality#/total}/count/instantaneous/suspended
/threads{locality#
/worker-thread#}/count/instantaneous/suspended
/threads{locality#
/total}/count/instantaneous/terminated
/threads{locality#/worker-thread#}/count/instantaneous/terminated
/threads{locality#/allocator#}/count/objects
/threads{locality#/total}/count/stack-recycles
/threads{locality#
/total}/count/stack-unbinds
/threads{locality#/total}/idle-rate
/threads{locality#
/worker-thread#*}/idle-rate
hello world from OS-thread 0 on locality 0
�]0;sameer@cn167:bin�[sameer@cn167 bin]$ exit
exit

Script done on Mon 29 Oct 2012 05:34:23 PM PDT

@hkaiser
Copy link
Member

hkaiser commented Oct 30, 2012

Thanks Sameer. I think I understand what is going on. You're using tcsh which messes with the '{' and '}' in the countername. Could you try to quote the countername or otherwise escape the special characters, please?

@sameershende
Copy link
Author

Hi Hartmut,
Thanks for all your help! Yes, it works perfectly now.
- Sameer

On Oct 29, 2012, at 5:45 PM, Hartmut Kaiser notifications@github.com wrote:

Thanks Sameer. I think I understand what is going on. You're using tcsh which messes with the '{' and '}' in the countername. Could you try to quote the countername or otherwise escape the special characters, please?


Reply to this email directly or view it on GitHub.

@hkaiser hkaiser closed this as completed Oct 30, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants