Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

glibc munmap_chunk or free(): invalid pointer on SuperMIC #1759

Closed
zkhatami88 opened this issue Sep 23, 2015 · 4 comments
Closed

glibc munmap_chunk or free(): invalid pointer on SuperMIC #1759

zkhatami88 opened this issue Sep 23, 2015 · 4 comments
Labels
affecting: LSU category: core type: defect
Milestone
0.9.99

Comments

@zkhatami88
Copy link
Contributor

After finishing running code of hello_world example completely, I see this error:

Output on one node:

$ mpirun_rsh -ssh -np $PBS_NUM_NODES $(uniq $PBS_NODEFILE) /worka/work/zahra/zahra/hpx/bin/hello_world 
hello world from OS-thread 3 on locality 0
hello world from OS-thread 5 on locality 0
hello world from OS-thread 8 on locality 0
hello world from OS-thread 18 on locality 0
hello world from OS-thread 2 on locality 0
hello world from OS-thread 14 on locality 0
hello world from OS-thread 17 on locality 0
hello world from OS-thread 1 on locality 0
hello world from OS-thread 6 on locality 0
hello world from OS-thread 7 on locality 0
hello world from OS-thread 4 on locality 0
hello world from OS-thread 11 on locality 0
hello world from OS-thread 19 on locality 0
hello world from OS-thread 15 on locality 0
hello world from OS-thread 12 on locality 0
hello world from OS-thread 9 on locality 0
hello world from OS-thread 10 on locality 0
hello world from OS-thread 16 on locality 0
hello world from OS-thread 13 on locality 0
hello world from OS-thread 0 on locality 0
*** glibc detected *** /worka/work/zahra/zahra/hpx/bin/hello_world: munmap_chunk(): invalid pointer: 0x0000000000417620 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d05675e66]
/worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0(+0x3b57c)[0x2ae7764e857c]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx4util15reinit_destructEv+0x4a)[0x2ae775ee732a]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx7runtimeD2Ev+0x1d1)[0x2ae775b85ba1]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED2Ev+0x14d)[0x2ae775b27ced]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED0Ev+0x9)[0x2ae775b27e29]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(+0x19ffcc)[0x2ae775b4efcc]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx6detail12run_or_startERKNS_4util8functionIFiRN5boost15program_options13variables_mapEEvvEERKNS4_19options_descriptionEiPPcRKSt6vectorISsSaISsEERKNS2_IFvvEvvEESO_NS_12runtime_modeEb+0x258)[0x2ae775b4fa98]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x459dd6]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x45a1ca]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d0561ed5d]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x40c41d]
======= Memory map: ========
00400000-00492000 r-xp 00000000 a7b:38b50 144501525115397791             /worka/work/zahra/zahra/hpx/bin/hello_world
00692000-00693000 rw-p 00092000 a7b:38b50 144501525115397791             /worka/work/zahra/zahra/hpx/bin/hello_world
00693000-00697000 rw-p 00000000 00:00 0 
00876000-008f8000 rw-p 00000000 00:00 0                                  [heap]
32b9600000-32b9682000 r-xp 00000000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9682000-32b9881000 ---p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9881000-32b9882000 r--p 00081000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9882000-32b9883000 rw-p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
3d05200000-3d05220000 r-xp 00000000 08:01 7471154                        /lib64/ld-2.12.so
3d0541f000-3d05420000 r--p 0001f000 08:01 7471154                        /lib64/ld-2.12.so
3d05420000-3d05421000 rw-p 00020000 08:01 7471154                        /lib64/ld-2.12.so
3d05421000-3d05422000 rw-p 00000000 00:00 0 
3d05600000-3d0578a000 r-xp 00000000 08:01 7471160                        /lib64/libc-2.12.so
3d0578a000-3d0598a000 ---p 0018a000 08:01 7471160                        /lib64/libc-2.12.so
3d0598a000-3d0598e000 r--p 0018a000 08:01 7471160                        /lib64/libc-2.12.so
3d0598e000-3d0598f000 rw-p 0018e000 08:01 7471160                        /lib64/libc-2.12.so
3d0598f000-3d05994000 rw-p 00000000 00:00 0 
3d05a00000-3d05a02000 r-xp 00000000 08:01 7471172                        /lib64/libdl-2.12.so
3d05a02000-3d05c02000 ---p 00002000 08:01 7471172                        /lib64/libdl-2.12.so
3d05c02000-3d05c03000 r--p 00002000 08:01 7471172                        /lib64/libdl-2.12.so
3d05c03000-3d05c04000 rw-p 00003000 08:01 7471172                        /lib64/libdl-2.12.so
3d05e00000-3d05e17000 r-xp 00000000 08:01 7471165                        /lib64/libpthread-2.12.so
3d05e17000-3d06017000 ---p 00017000 08:01 7471165                        /lib64/libpthread-2.12.so
3d06017000-3d06018000 r--p 00017000 08:01 7471165                        /lib64/libpthread-2.12.so
3d06018000-3d06019000 rw-p 00018000 08:01 7471165                        /lib64/libpthread-2.12.so
3d06019000-3d0601d000 rw-p 00000000 00:00 0 
3d06a00000-3d06a07000 r-xp 00000000 08:01 7471168                        /lib64/librt-2.12.so
3d06a07000-3d06c06000 ---p 00007000 08:01 7471168                        /lib64/librt-2.12.so
3d06c06000-3d06c07000 r--p 00006000 08:01 7471168                        /lib64/librt-2.12.so
3d06c07000-3d06c08000 rw-p 00007000 08:01 7471168                        /lib64/librt-2.12.so
2ae7759ad000-2ae7759af000 rw-p 00000000 00:00 0 
2ae7759af000-2ae776234000 r-xp 00000000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2ae776234000-2ae776434000 ---p 00885000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2ae776434000-2ae77647c000 rw-p 00885000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2ae77647c000-2ae7764ad000 rw-p 00000000 00:00 0 
2ae7764ad000-2ae77654e000 r-xp 00000000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2ae77654e000-2ae77674d000 ---p 000a1000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2ae77674d000-2ae776757000 rw-p 000a0000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2ae776757000-2ae77675a000 rw-p 00000000 00:00 0 
2ae776773000-2ae776774000 rw-p 00000000 00:00 0 
2ae776774000-2ae77677a000 r-xp 00000000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2ae77677a000-2ae77697a000 ---p 00006000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2ae77697a000-2ae77697b000 rw-p 00006000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2ae77697b000-2ae77698b000 r-xp 00000000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2ae77698b000-2ae776b8b000 ---p 00010000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2ae776b8b000-2ae776b8c000 rw-p 00010000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2ae776b8c000-2ae776b8d000 rw-p 00000000 00:00 0 
2ae776b8d000-2ae776ba4000 r-xp 00000000 00:14 188882391                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_filesystem.so.1.58.0
3ae776ba4000-2ae776da3000 ---p 00017000 00:14 188882391                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_filesystem.so.1.58.0[smic019:mpispawn_0][child_handler] MPI process (rank: 0, pid: 107600) terminated with signal 6 -> abort job
[smic019:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node smic019 aborted: MPI process error (1)

Output on more than one node:

$ mpirun_rsh -ssh -np $PBS_NUM_NODES $(uniq $PBS_NODEFILE) /worka/work/zahra/zahra/hpx/bin/hello_world 
hello world from OS-thread 1 on locality 0
hello world from OS-thread 6 on locality 0
hello world from OS-thread 2 on locality 0
hello world from OS-thread 17 on locality 0
hello world from OS-thread 0 on locality 1
hello world from OS-thread 16 on locality 0
hello world from OS-thread 13 on locality 0
hello world from OS-thread 18 on locality 0
hello world from OS-thread 4 on locality 0
hello world from OS-thread 9 on locality 0
hello world from OS-thread 10 on locality 0
hello world from OS-thread 12 on locality 0
hello world from OS-thread 15 on locality 0
hello world from OS-thread 7 on locality 0
hello world from OS-thread 11 on locality 0
hello world from OS-thread 19 on locality 0
hello world from OS-thread 14 on locality 0
hello world from OS-thread 3 on locality 0
hello world from OS-thread 0 on locality 0
hello world from OS-thread 5 on locality 0
hello world from OS-thread 8 on locality 0
*** glibc detected *** /worka/work/zahra/zahra/hpx/bin/hello_world: free(): invalid pointer: 0x00002b8d0c026c48 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3cb0c75e66]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx4util15reinit_destructEv+0x4a)[0x2b8c717db32a]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx7runtimeD2Ev+0x1d1)[0x2b8c71479ba1]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED2Ev+0x14d)[0x2b8c7141bced]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED0Ev+0x9)[0x2b8c7141be29]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(+0x19ffcc)[0x2b8c71442fcc]
/worka/work/zahra/zahra/hpx/lib/libhpx.so.0(_ZN3hpx6detail12run_or_startERKNS_4util8functionIFiRN5boost15program_options13variables_mapEEvvEERKNS4_19options_descriptionEiPPcRKSt6vectorISsSaISsEERKNS2_IFvvEvvEESO_NS_12runtime_modeEb+0x258)[0x2b8c71443a98]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x459dd6]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x45a1ca]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3cb0c1ed5d]
/worka/work/zahra/zahra/hpx/bin/hello_world[0x40c41d]
======= Memory map: ========
00400000-00492000 r-xp 00000000 a7b:38b50 144501525115397791             /worka/work/zahra/zahra/hpx/bin/hello_world
00692000-00693000 rw-p 00092000 a7b:38b50 144501525115397791             /worka/work/zahra/zahra/hpx/bin/hello_world
00693000-00697000 rw-p 00000000 00:00 0 
010f5000-01177000 rw-p 00000000 00:00 0                                  [heap]
32b9600000-32b9682000 r-xp 00000000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9682000-32b9881000 ---p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9881000-32b9882000 r--p 00081000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9882000-32b9883000 rw-p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
3cb0800000-3cb0820000 r-xp 00000000 08:01 3670066                        /lib64/ld-2.12.so
3cb0a1f000-3cb0a20000 r--p 0001f000 08:01 3670066                        /lib64/ld-2.12.so
3cb0a20000-3cb0a21000 rw-p 00020000 08:01 3670066                        /lib64/ld-2.12.so
3cb0a21000-3cb0a22000 rw-p 00000000 00:00 0 
3cb0c00000-3cb0d8a000 r-xp 00000000 08:01 3670072                        /lib64/libc-2.12.so
3cb0d8a000-3cb0f8a000 ---p 0018a000 08:01 3670072                        /lib64/libc-2.12.so
3cb0f8a000-3cb0f8e000 r--p 0018a000 08:01 3670072                        /lib64/libc-2.12.so
3cb0f8e000-3cb0f8f000 rw-p 0018e000 08:01 3670072                        /lib64/libc-2.12.so
3cb0f8f000-3cb0f94000 rw-p 00000000 00:00 0 
3cb1000000-3cb1002000 r-xp 00000000 08:01 3670084                        /lib64/libdl-2.12.so
3cb1002000-3cb1202000 ---p 00002000 08:01 3670084                        /lib64/libdl-2.12.so
3cb1202000-3cb1203000 r--p 00002000 08:01 3670084                        /lib64/libdl-2.12.so
3cb1203000-3cb1204000 rw-p 00003000 08:01 3670084                        /lib64/libdl-2.12.so
3cb1400000-3cb1417000 r-xp 00000000 08:01 3670077                        /lib64/libpthread-2.12.so
3cb1417000-3cb1617000 ---p 00017000 08:01 3670077                        /lib64/libpthread-2.12.so
3cb1617000-3cb1618000 r--p 00017000 08:01 3670077                        /lib64/libpthread-2.12.so
3cb1618000-3cb1619000 rw-p 00018000 08:01 3670077                        /lib64/libpthread-2.12.so
3cb1619000-3cb161d000 rw-p 00000000 00:00 0 
3cb2000000-3cb2007000 r-xp 00000000 08:01 3670080                        /lib64/librt-2.12.so
3cb2007000-3cb2206000 ---p 00007000 08:01 3670080                        /lib64/librt-2.12.so
3cb2206000-3cb2207000 r--p 00006000 08:01 3670080                        /lib64/librt-2.12.so
3cb2207000-3cb2208000 rw-p 00007000 08:01 3670080                        /lib64/librt-2.12.so
2b8c712a1000-2b8c712a3000 rw-p 00000000 00:00 0 
2b8c712a3000-2b8c71b28000 r-xp 00000000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2b8c71b28000-2b8c71d28000 ---p 00885000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2b8c71d28000-2b8c71d70000 rw-p 00885000 a7b:38b50 144501525115397155     /worka/work/zahra/zahra/hpx/lib/libhpx.so.0.9.11
2b8c71d70000-2b8c71da1000 rw-p 00000000 00:00 0 
2b8c71da1000-2b8c71e42000 r-xp 00000000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2b8c71e42000-2b8c72041000 ---p 000a1000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2b8c72041000-2b8c7204b000 rw-p 000a0000 a7b:38b50 144501525115397532     /worka/work/zahra/zahra/hpx/lib/libhpx_iostreams.so.0.9.11
2b8c7204b000-2b8c7204e000 rw-p 00000000 00:00 0 
2b8c72067000-2b8c72068000 rw-p 00000000 00:00 0 
2b8c72068000-2b8c7206e000 r-xp 00000000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2b8c7206e000-2b8c7226e000 ---p 00006000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2b8c7226e000-2b8c7226f000 rw-p 00006000 00:14 188882309                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2b8c7226f000-2b8c7227f000 r-xp 00000000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2b8c7227f000-2b8c7247f000 ---p 00010000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2b8c7247f000-2b8c72480000 rw-p 00010000 00:14 188882366                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2b8c72480000-2b8c72481000 rw-p 00000000 00:00 0 
2b8c72481000-2b8c72498000 r-xp 00000000 00:14 188882391                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_filesystem.so.1.58.0
2b8c72498000-2b8c72697000 ---p 00017000 00:14 188882391                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_filesystem.so.1.58.0
2b8c72697000-2b8c72698000 rw-p 00016000 00:14 188882391                  /home/zahra/zahra/boost_1_58_0/stage/lib/libboost_filesystem.so.1.58.0[smic018:mpispawn_0][child_handler] MPI process (rank: 0, pid: 120435) terminated with signal 6 -> abort job
[smic018:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node smic018 aborted: MPI process error (1)

Memory Allocator: Custom (MVAPICH2)
HPX Configuration Command:

cmake -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc -DCMAKE_BUILD_TYPE=Release -DBOOST_ROOT=/home/zahra/zahra/boost_1_58_0 -DHPX_WITH_MALLOC=custom -DHPX_WITH_PARCELPORT_MPI=True -DHPX_WITH_PARCELPORT_TCP=False -DHPX_WITH_PARCELPORT_IBVERBS=False -DHPX_WITH_EXAMPLES=True /home/zahra/zahra/repo

HPX Print Info:

Static configuration:
---------------------
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_TCP=OFF
  HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
  HPX_HAVE_PARCELPORT_IPC=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
  HPX_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_HAVE_MALLOC=custom
  HPX_PREFIX (configured)=/worka/work/zahra/zahra/hpx
  HPX_PREFIX=/worka/work/zahra/zahra/hpx

Runtime configuration:
----------------------
  {mode}: console
  {scheduler}: local-priority
  {os-threads}: 20
  {cores}: 20

Version Info:

HPX - High Performance ParalleX
A general purpose parallel C++ runtime system for             distributed applications
of any scale.

Copyright (c) 2007-2015, The STE||AR Group,
http://stellar-group.org, email:hpx-users@stellar.cct.lsu.edu

Distributed under the Boost Software License,              Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

Versions:
  HPX: V0.9.11-trunk (AGAS: V3.0), Git: 45a676d516
  Boost: V1.58.0
  Hwloc: V1.10.0
  MPI: MPICH V3.1, MPI V3.0

Build:
  Type: release
  Date: Sep 23 2015 09:44:21
  Platform: linux
  Compiler: GNU C++ version 4.9.0
  Standard Library: GNU libstdc++ version 20140422
  Allocator: custom
@hkaiser hkaiser added this to the 0.9.11 milestone Sep 23, 2015
@sithhell
Copy link
Member

sithhell commented Sep 23, 2015 via email

@parsa
Copy link
Contributor

parsa commented Sep 23, 2015

Runs fine with jemallloc
With system allocator we get the same error messages.
One node:

$ mpirun_rsh -ssh -np $PBS_NUM_NODES $(uniq $PBS_NODEFILE) /worka/work/parsa/t1000/hpx_salloc/bin/hello_world 
hello world from OS-thread 1 on locality 0
hello world from OS-thread 4 on locality 0
hello world from OS-thread 11 on locality 0
hello world from OS-thread 17 on locality 0
hello world from OS-thread 12 on locality 0
hello world from OS-thread 3 on locality 0
hello world from OS-thread 6 on locality 0
hello world from OS-thread 7 on locality 0
hello world from OS-thread 5 on locality 0
hello world from OS-thread 9 on locality 0
hello world from OS-thread 13 on locality 0
hello world from OS-thread 10 on locality 0
hello world from OS-thread 15 on locality 0
hello world from OS-thread 8 on locality 0
hello world from OS-thread 2 on locality 0
hello world from OS-thread 16 on locality 0
hello world from OS-thread 18 on locality 0
hello world from OS-thread 19 on locality 0
hello world from OS-thread 14 on locality 0
hello world from OS-thread 0 on locality 0
*** glibc detected *** /worka/work/parsa/t1000/hpx_salloc/bin/hello_world: munmap_chunk(): invalid pointer: 0x0000000000417630 ***
======= Backtrace: =========
/lib64/libc.so.6[0x31f3275e66]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx_iostreams.so.0(+0x3b58c)[0x2baa1cea158c]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(_ZN3hpx4util15reinit_destructEv+0x4a)[0x2baa1c8a035a]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(_ZN3hpx7runtimeD2Ev+0x1d1)[0x2baa1c53ebb1]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED2Ev+0x14d)[0x2baa1c4e0cfd]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(_ZN3hpx12runtime_implINS_7threads8policies30local_priority_queue_schedulerIN5boost5mutexENS2_13lockfree_fifoES6_NS2_13lockfree_lifoEEEED0Ev+0x9)[0x2baa1c4e0e39]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(+0x19ffdc)[0x2baa1c507fdc]
/worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0(_ZN3hpx6detail12run_or_startERKNS_4util8functionIFiRN5boost15program_options13variables_mapEEvvEERKNS4_19options_descriptionEiPPcRKSt6vectorISsSaISsEERKNS2_IFvvEvvEESO_NS_12runtime_modeEb+0x258)[0x2baa1c508aa8]
/worka/work/parsa/t1000/hpx_salloc/bin/hello_world[0x459de6]
/worka/work/parsa/t1000/hpx_salloc/bin/hello_world[0x45a1da]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x31f321ed5d]
/worka/work/parsa/t1000/hpx_salloc/bin/hello_world[0x40c42d]
======= Memory map: ========
00400000-00492000 r-xp 00000000 a7b:38b50 144509141585570700             /worka/work/parsa/t1000/hpx_salloc/bin/hello_world
00692000-00693000 rw-p 00092000 a7b:38b50 144509141585570700             /worka/work/parsa/t1000/hpx_salloc/bin/hello_world
00693000-00697000 rw-p 00000000 00:00 0 
0168b000-0170d000 rw-p 00000000 00:00 0                                  [heap]
31f2e00000-31f2e20000 r-xp 00000000 08:01 4849714                        /lib64/ld-2.12.so
31f301f000-31f3020000 r--p 0001f000 08:01 4849714                        /lib64/ld-2.12.so
31f3020000-31f3021000 rw-p 00020000 08:01 4849714                        /lib64/ld-2.12.so
31f3021000-31f3022000 rw-p 00000000 00:00 0 
31f3200000-31f338a000 r-xp 00000000 08:01 4849720                        /lib64/libc-2.12.so
31f338a000-31f358a000 ---p 0018a000 08:01 4849720                        /lib64/libc-2.12.so
31f358a000-31f358e000 r--p 0018a000 08:01 4849720                        /lib64/libc-2.12.so
31f358e000-31f358f000 rw-p 0018e000 08:01 4849720                        /lib64/libc-2.12.so
31f358f000-31f3594000 rw-p 00000000 00:00 0 
31f3600000-31f3602000 r-xp 00000000 08:01 4849732                        /lib64/libdl-2.12.so
31f3602000-31f3802000 ---p 00002000 08:01 4849732                        /lib64/libdl-2.12.so
31f3802000-31f3803000 r--p 00002000 08:01 4849732                        /lib64/libdl-2.12.so
31f3803000-31f3804000 rw-p 00003000 08:01 4849732                        /lib64/libdl-2.12.so
31f3a00000-31f3a17000 r-xp 00000000 08:01 4849725                        /lib64/libpthread-2.12.so
31f3a17000-31f3c17000 ---p 00017000 08:01 4849725                        /lib64/libpthread-2.12.so
31f3c17000-31f3c18000 r--p 00017000 08:01 4849725                        /lib64/libpthread-2.12.so
31f3c18000-31f3c19000 rw-p 00018000 08:01 4849725                        /lib64/libpthread-2.12.so
31f3c19000-31f3c1d000 rw-p 00000000 00:00 0 
31f4600000-31f4607000 r-xp 00000000 08:01 4849728                        /lib64/librt-2.12.so
31f4607000-31f4806000 ---p 00007000 08:01 4849728                        /lib64/librt-2.12.so
31f4806000-31f4807000 r--p 00006000 08:01 4849728                        /lib64/librt-2.12.so
31f4807000-31f4808000 rw-p 00007000 08:01 4849728                        /lib64/librt-2.12.so
32b9600000-32b9682000 r-xp 00000000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9682000-32b9881000 ---p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9881000-32b9882000 r--p 00081000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
32b9882000-32b9883000 rw-p 00082000 00:14 37359782                       /home/packages/python/2.7.7-anaconda/lib/libm-2.5.so
2baa1c366000-2baa1c368000 rw-p 00000000 00:00 0 
2baa1c368000-2baa1cbed000 r-xp 00000000 a7b:38b50 144509141585570055     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0.9.11
2baa1cbed000-2baa1cded000 ---p 00885000 a7b:38b50 144509141585570055     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0.9.11
2baa1cded000-2baa1ce35000 rw-p 00885000 a7b:38b50 144509141585570055     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx.so.0.9.11
2baa1ce35000-2baa1ce66000 rw-p 00000000 00:00 0 
2baa1ce66000-2baa1cf07000 r-xp 00000000 a7b:38b50 144509141585570433     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx_iostreams.so.0.9.11
2baa1cf07000-2baa1d106000 ---p 000a1000 a7b:38b50 144509141585570433     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx_iostreams.so.0.9.11
2baa1d106000-2baa1d110000 rw-p 000a0000 a7b:38b50 144509141585570433     /worka/work/parsa/t1000/hpx_salloc/lib/libhpx_iostreams.so.0.9.11
2baa1d110000-2baa1d113000 rw-p 00000000 00:00 0 
2baa1d12c000-2baa1d12d000 rw-p 00000000 00:00 0 
2baa1d12d000-2baa1d133000 r-xp 00000000 00:14 188882309                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2baa1d133000-2baa1d333000 ---p 00006000 00:14 188882309                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2baa1d333000-2baa1d334000 rw-p 00006000 00:14 188882309                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_chrono.so.1.58.0
2baa1d334000-2baa1d344000 r-xp 00000000 00:14 188882366                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2baa1d344000-2baa1d544000 ---p 00010000 00:14 188882366                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0
2baa1d544000-2baa1d545000 rw-p 00010000 00:14 188882366                  /home/parsa/t1000/boost_1_58_0/stage/lib/libboost_date_time.so.1.58.0[smic002:mpispawn_0][child_handler] MPI process (rank: 0, pid: 90020) terminated with signal 6 -> abort job
[smic002:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node smic002 aborted: MPI process error (1)

Config

$ mpirun_rsh -ssh -np $PBS_NUM_NODES $(uniq $PBS_NODEFILE) /worka/work/parsa/t1000/hpx_salloc/bin/hello_world --hpx:info
Static configuration:
---------------------
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_HAVE_PARCELPORT_TCP=OFF
  HPX_HAVE_PARCELPORT_MPI=ON (MPICH V3.1, MPI V3.0)
  HPX_HAVE_PARCELPORT_IPC=OFF
  HPX_HAVE_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
  HPX_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_HAVE_MALLOC=system
  HPX_PREFIX (configured)=/worka/work/parsa/t1000/hpx_salloc
  HPX_PREFIX=/worka/work/parsa/t1000/hpx_salloc

Runtime configuration:
----------------------
  {mode}: console
  {scheduler}: local-priority
  {os-threads}: 20
  {cores}: 20

@hkaiser
Copy link
Member

hkaiser commented Nov 7, 2015

This is clearly caused by an allocator mismatch between HPX and the application. A chunk of memory is attempted to be deallocated by a different allocator than it was allocated.

Any idea what can be done to either circumvent, but at least diagnose this kind of issues?

@hkaiser hkaiser modified the milestones: 0.9.11, 0.9.12 Nov 12, 2015
@sithhell
Copy link
Member

This is a problem with gcc 4.9.0, any other version (even gcc 4.9.3) should work fine!
Please reopen if the problem still exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affecting: LSU category: core type: defect
Projects
None yet
Milestone
0.9.99
Development

No branches or pull requests
4 participants
@sithhell @parsa @hkaiser @zkhatami88