ci: Set openmpi and openmp parameters to make an efficient use of the CI #3908

vicentebolea · 2023-11-08T18:04:12Z

anagainaru · 2023-11-08T18:34:32Z

Is this going to make the testing (and thus the CI) slower? Do we really want to keep the BP3 testing? Could we maybe separate it to only test BP3 in the nightly builds?

eisenhauer · 2023-11-08T18:43:38Z

I'm curious if this really fixes it and why. This isn't big data. If one of the compressors is taking more than 2 minutes to do something with it, that seems like a non-viable compressor.

vicentebolea · 2023-11-13T18:33:10Z

It is true, some of the instances of those tests takes <10s and others about 3 minutes. I was able to replicate this in my machine and I found that this only happens when we use same or more mpi tasks as cores available.

Also here is the Backtrace which seems to show that the MPI tasks are not synchronized, some of them are waiting at MPI_GATHER and some others in MPI_BARRIER. I wonder if it can this be that mgard is calling this MPI routines in its own? @eisenhauer

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f91d60648c6 in epoll_wait () from /lib64/libc.so.6
#0  0x00007f91d60648c6 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000fa635f in ?? ()
#2  0x0000000000f96ed2 in ?? ()
#3  0x000000000056196b in ?? ()
#4  0x00000000004f28f5 in ?? ()
#5  0x00000000004f2d14 in ?? ()
#6  0x0000000000449bab in ?? ()
#7  0x000000000044a598 in ?? ()
#8  0x000000000044b117 in ?? ()
#9  0x0000000000413755 in ?? ()
#10 0x00007f91d5f8124d in __libc_start_main () from /lib64/libc.so.6
#11 0x0000000000430e95 in ?? ()
#12 0x00007ffd31ff4e68 in ?? ()
#13 0x000000000000001c in ?? ()
#14 0x0000000000000006 in ?? ()
#15 0x00007ffd31ff6ce0 in ?? ()
#16 0x00007ffd31ff6ce6 in ?? ()
#17 0x00007ffd31ff6cfa in ?? ()
#18 0x00007ffd31ff6cfd in ?? ()
#19 0x00007ffd31ff6d01 in ?? ()
#20 0x00007ffd31ff6d04 in ?? ()
#21 0x0000000000000000 in ?? ()
[Inferior 1 (process 13898) detached]
[New LWP 13995]
[New LWP 13996]
[New LWP 13997]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fb6244f5a09 in poll () from /lib64/libc.so.6
#0  0x00007fb6244f5a09 in poll () from /lib64/libc.so.6
#1  0x00007fb62463057d in poll_dispatch (base=0x5596d96995e0, tv=<optimized out>) at poll.c:182
#2  0x00007fb624627a65 in event_base_loop (base=0x5596d96995e0, flags=1) at event.c:1992
#3  0x00005596d90bc6a1 in orterun ()
#4  0x00007fb62441f24d in __libc_start_main () from /lib64/libc.so.6
#5  0x00005596d90bc40a in _start () at ../sysdeps/x86_64/start.S:120
[Inferior 1 (process 13994) detached]
[New LWP 14032]
[New LWP 14036]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f7a14877a09 in poll () from /lib64/libc.so.6
#0  0x00007f7a14877a09 in poll () from /lib64/libc.so.6
#1  0x00007f7a142ac57d in poll_dispatch (base=0x204f0a0, tv=<optimized out>) at poll.c:182
#2  0x00007f7a142a3a65 in event_base_loop (base=0x204f0a0, flags=2) at event.c:1992
#3  0x00007f7a1457ad96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f7a1457aeeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f7a145812e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f7a15e9c100 in mca_pml_ob1_recv () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f7a15d6adf6 in ompi_coll_base_gather_intra_binomial () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f7a15d7f511 in ompi_coll_tuned_gather_intra_dec_fixed () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f7a15d27542 in PMPI_Gather () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#10 0x00007f7a163650fa in adios2::helper::CommImplMPI::Gather (this=0x229ac50, sendbuf=0x7ffcf102e150, sendcount=1, sendtype=adios2::helper::CommImpl::Datatype::UnsignedLong, recvbuf=0x22c1f30, recvcount=1, recvtype=adios2::helper::CommImpl::Datatype::UnsignedLong, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:336
#11 0x00007f7a153a9905 in adios2::helper::Comm::Gather<unsigned long, unsigned long> (this=0x229bf08, sendbuf=0x7ffcf102e150, sendcount=1, recvbuf=0x22c1f30, recvcount=1, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.inl:211
#12 0x00007f7a155e6a14 in adios2::helper::Comm::GatherArrays<unsigned long> (this=0x229bf08, source=0x7ffcf102e150, sourceCount=1, destination=0x22c1f30, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:30
#13 0x00007f7a155e62a3 in adios2::helper::Comm::GatherValues<unsigned long> (this=0x229bf08, source=296, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:48
#14 0x00007f7a156a1ece in adios2::helper::Comm::GathervVectors<char> (this=0x229bf08, in=std::vector of length 296, capacity 296 = {...}, out=std::vector of length 0, capacity 0, position=@0x229c0f0: 0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:83
#15 0x00007f7a1568726a in adios2::format::BP3Serializer::AggregateCollectiveMetadataIndices (this=0x229bf60, comm=..., bufferSTL=...) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:544
#16 0x00007f7a15686ab2 in adios2::format::BP3Serializer::AggregateCollectiveMetadata (this=0x229bf60, comm=..., bufferSTL=..., inMetadataBuffer=true) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:196
#17 0x00007f7a1555ea4a in adios2::core::engine::BP3Writer::WriteCollectiveMetadataFile (this=0x229beb0, isFinal=false) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:329
#18 0x00007f7a1555e8d0 in adios2::core::engine::BP3Writer::Flush (this=0x229beb0, transportIndex=-1) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:119
#19 0x00007f7a1555e7c3 in adios2::core::engine::BP3Writer::EndStep (this=0x229beb0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:107
#20 0x00007f7a16148a9b in adios2::Engine::EndStep (this=0x7ffcf102f620) at ../ADIOS2/bindings/CXX11/adios2/cxx11/Engine.cpp:113
#21 0x00000000004266f8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x2259540) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:109
#22 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x2259540, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#23 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x2259540, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#24 0x000000000044ecc9 in testing::Test::Run (this=0x2259540) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#25 0x000000000044f869 in testing::TestInfo::Run (this=0x2259130) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#26 0x00000000004500fe in testing::TestSuite::Run (this=0x2253970) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#27 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x200b300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#28 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x200b300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#29 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x200b300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#30 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#31 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#32 0x0000000000428c52 in main (argc=2, argv=0x7ffcf102fee8) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 13998) detached]
[New LWP 14013]
[New LWP 14015]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fe8abd28a09 in poll () from /lib64/libc.so.6
#0  0x00007fe8abd28a09 in poll () from /lib64/libc.so.6
#1  0x00007fe8ab75d57d in poll_dispatch (base=0x1a090a0, tv=<optimized out>) at poll.c:182
#2  0x00007fe8ab754a65 in event_base_loop (base=0x1a090a0, flags=2) at event.c:1992
#3  0x00007fe8aba2bd96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007fe8aba2beeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007fe8aba322e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007fe8ad351351 in mca_pml_ob1_send () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007fe8ad22264f in ompi_coll_base_barrier_intra_recursivedoubling () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007fe8ad1c7f98 in PMPI_Barrier () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007fe8ad815a5d in adios2::helper::CommImplMPI::Barrier (this=0x1c546b0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:276
#10 0x00007fe8ac9c4569 in adios2::helper::Comm::Barrier (this=0x1c55a48, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:75
#11 0x00007fe8acd67c72 in adios2::transportman::TransportMan::MkDirsBarrier (this=0x1c55f80, fileNames=std::vector of length 32, capacity 32, parametersVector=std::vector of length 1, capacity 1 = {...}, nodeLocal=false) at ../ADIOS2/source/adios2/toolkit/transportman/TransportMan.cpp:104
#12 0x00007fe8aca1016a in adios2::core::engine::BP3Writer::InitTransports (this=0x1c559f0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:195
#13 0x00007fe8aca0e31a in adios2::core::engine::BP3Writer::Init (this=0x1c559f0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:133
#14 0x00007fe8aca0e17f in adios2::core::engine::BP3Writer::BP3Writer (this=0x1c559f0, io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:34
#15 0x00007fe8ac927fc7 in __gnu_cxx::new_allocator<adios2::core::engine::BP3Writer>::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffff6eacc60, __p=0x1c559f0, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/ext/new_allocator.h:156
#16 0x00007fe8ac927d2c in std::allocator_traits<std::allocator<adios2::core::engine::BP3Writer> >::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __p=0x1c559f0, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/alloc_traits.h:512
#17 0x00007fe8ac927a60 in std::_Sp_counted_ptr_inplace<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x1c559e0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:551
#18 0x00007fe8ac9277f5 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffff6eacee8, __p=@0x7ffff6eacee0: 0x0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:683
#19 0x00007fe8ac9276d6 in std::__shared_ptr<adios2::core::engine::BP3Writer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffff6eacee0, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:1376
#20 0x00007fe8ac92762c in std::shared_ptr<adios2::core::engine::BP3Writer>::shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffff6eacee0, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:408
#21 0x00007fe8ac927565 in std::allocate_shared<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:861
#22 0x00007fe8ac927443 in std::make_shared<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:877
#23 0x00007fe8ac90d6e5 in adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer> (io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.h:473
#24 0x00007fe8ad813fc4 in std::__invoke_impl<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__f=@0x19c2548: 0x7fe8ac90d690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:60
#25 0x00007fe8ad813eca in std::__invoke_r<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__fn=@0x19c2548: 0x7fe8ac90d690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:141
#26 0x00007fe8ad813d4a in std::_Function_handler<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), std::shared_ptr<adios2::core::Engine> (*)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::_M_invoke(std::_Any_data const&, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode&&, adios2::helper::Comm&&) (__functor=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:291
#27 0x00007fe8ac921d30 in std::function<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::operator()(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm) const (this=0x19c2548, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:622
#28 0x00007fe8ac8e64cb in adios2::core::IO::Open (this=0x1c53180, name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.cpp:643
#29 0x00007fe8ac8e6b12 in adios2::core::IO::Open (this=0x1c53180, name=, mode=adios2::Mode::Write) at ../ADIOS2/source/adios2/core/IO.cpp:665
#30 0x00007fe8ad65efc6 in adios2::IO::Open (this=0x7ffff6eae710, name=, mode=adios2::Mode::Write) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#31 0x00000000004258d8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x1c13570) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:65
#32 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x1c13570, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#33 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x1c13570, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#34 0x000000000044ecc9 in testing::Test::Run (this=0x1c13570) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#35 0x000000000044f869 in testing::TestInfo::Run (this=0x1c13160) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#36 0x00000000004500fe in testing::TestSuite::Run (this=0x1c0d9a0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#37 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x19c5300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#38 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x19c5300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#39 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x19c5300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#40 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#41 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#42 0x0000000000428c52 in main (argc=2, argv=0x7ffff6eaef88) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 13999) detached]
[New LWP 14031]
[New LWP 14034]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fba01286a09 in poll () from /lib64/libc.so.6
#0  0x00007fba01286a09 in poll () from /lib64/libc.so.6
#1  0x00007fba00cbb57d in poll_dispatch (base=0x99d0a0, tv=<optimized out>) at poll.c:182
#2  0x00007fba00cb2a65 in event_base_loop (base=0x99d0a0, flags=2) at event.c:1992
#3  0x00007fba00f89d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007fba00f89eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007fba00f902e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007fba028ab100 in mca_pml_ob1_recv () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007fba02779df6 in ompi_coll_base_gather_intra_binomial () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007fba0278e511 in ompi_coll_tuned_gather_intra_dec_fixed () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007fba02736542 in PMPI_Gather () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#10 0x00007fba02d740fa in adios2::helper::CommImplMPI::Gather (this=0xbe8840, sendbuf=0x7fffe3697f70, sendcount=1, sendtype=adios2::helper::CommImpl::Datatype::UnsignedLong, recvbuf=0x0, recvcount=1, recvtype=adios2::helper::CommImpl::Datatype::UnsignedLong, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:336
#11 0x00007fba01db8905 in adios2::helper::Comm::Gather<unsigned long, unsigned long> (this=0xbe9c18, sendbuf=0x7fffe3697f70, sendcount=1, recvbuf=0x0, recvcount=1, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.inl:211
#12 0x00007fba01ff5a14 in adios2::helper::Comm::GatherArrays<unsigned long> (this=0xbe9c18, source=0x7fffe3697f70, sourceCount=1, destination=0x0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:30
#13 0x00007fba01ff52a3 in adios2::helper::Comm::GatherValues<unsigned long> (this=0xbe9c18, source=296, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:48
#14 0x00007fba020b0ece in adios2::helper::Comm::GathervVectors<char> (this=0xbe9c18, in=std::vector of length 296, capacity 296 = {...}, out=std::vector of length 0, capacity 0, position=@0xbe9e00: 0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:83
#15 0x00007fba0209626a in adios2::format::BP3Serializer::AggregateCollectiveMetadataIndices (this=0xbe9c70, comm=..., bufferSTL=...) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:544
#16 0x00007fba02095ab2 in adios2::format::BP3Serializer::AggregateCollectiveMetadata (this=0xbe9c70, comm=..., bufferSTL=..., inMetadataBuffer=true) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:196
#17 0x00007fba01f6da4a in adios2::core::engine::BP3Writer::WriteCollectiveMetadataFile (this=0xbe9bc0, isFinal=false) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:329
#18 0x00007fba01f6d8d0 in adios2::core::engine::BP3Writer::Flush (this=0xbe9bc0, transportIndex=-1) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:119
#19 0x00007fba01f6d7c3 in adios2::core::engine::BP3Writer::EndStep (this=0xbe9bc0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:107
#20 0x00007fba02b57a9b in adios2::Engine::EndStep (this=0x7fffe3699440) at ../ADIOS2/bindings/CXX11/adios2/cxx11/Engine.cpp:113
#21 0x00000000004266f8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0xba7530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:109
#22 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xba7530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#23 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xba7530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#24 0x000000000044ecc9 in testing::Test::Run (this=0xba7530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#25 0x000000000044f869 in testing::TestInfo::Run (this=0xba7120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#26 0x00000000004500fe in testing::TestSuite::Run (this=0xba1960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#27 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x959300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#28 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x959300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#29 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x959300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#30 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#31 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#32 0x0000000000428c52 in main (argc=2, argv=0x7fffe3699cf8) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14000) detached]
[New LWP 14028]
[New LWP 14029]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f73cee0ca09 in poll () from /lib64/libc.so.6
#0  0x00007f73cee0ca09 in poll () from /lib64/libc.so.6
#1  0x00007f73ce84157d in poll_dispatch (base=0x7b20a0, tv=<optimized out>) at poll.c:182
#2  0x00007f73ce838a65 in event_base_loop (base=0x7b20a0, flags=2) at event.c:1992
#3  0x00007f73ceb0fd96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f73ceb0feeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f73ceb162e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f73d0435351 in mca_pml_ob1_send () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f73d030664f in ompi_coll_base_barrier_intra_recursivedoubling () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f73d02abf98 in PMPI_Barrier () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f73d08f9a5d in adios2::helper::CommImplMPI::Barrier (this=0x9fd610, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:276
#10 0x00007f73cfaa8569 in adios2::helper::Comm::Barrier (this=0x9fe978, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:75
#11 0x00007f73cfe4bc72 in adios2::transportman::TransportMan::MkDirsBarrier (this=0x9feeb0, fileNames=std::vector of length 32, capacity 32, parametersVector=std::vector of length 1, capacity 1 = {...}, nodeLocal=false) at ../ADIOS2/source/adios2/toolkit/transportman/TransportMan.cpp:104
#12 0x00007f73cfaf416a in adios2::core::engine::BP3Writer::InitTransports (this=0x9fe920) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:195
#13 0x00007f73cfaf231a in adios2::core::engine::BP3Writer::Init (this=0x9fe920) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:133
#14 0x00007f73cfaf217f in adios2::core::engine::BP3Writer::BP3Writer (this=0x9fe920, io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:34
#15 0x00007f73cfa0bfc7 in __gnu_cxx::new_allocator<adios2::core::engine::BP3Writer>::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe6eebd0e0, __p=0x9fe920, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/ext/new_allocator.h:156
#16 0x00007f73cfa0bd2c in std::allocator_traits<std::allocator<adios2::core::engine::BP3Writer> >::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __p=0x9fe920, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/alloc_traits.h:512
#17 0x00007f73cfa0ba60 in std::_Sp_counted_ptr_inplace<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x9fe910, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:551
#18 0x00007f73cfa0b7f5 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe6eebd368, __p=@0x7ffe6eebd360: 0x0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:683
#19 0x00007f73cfa0b6d6 in std::__shared_ptr<adios2::core::engine::BP3Writer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe6eebd360, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:1376
#20 0x00007f73cfa0b62c in std::shared_ptr<adios2::core::engine::BP3Writer>::shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe6eebd360, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:408
#21 0x00007f73cfa0b565 in std::allocate_shared<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:861
#22 0x00007f73cfa0b443 in std::make_shared<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:877
#23 0x00007f73cf9f16e5 in adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer> (io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.h:473
#24 0x00007f73d08f7fc4 in std::__invoke_impl<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__f=@0x76b548: 0x7f73cf9f1690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:60
#25 0x00007f73d08f7eca in std::__invoke_r<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__fn=@0x76b548: 0x7f73cf9f1690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:141
#26 0x00007f73d08f7d4a in std::_Function_handler<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), std::shared_ptr<adios2::core::Engine> (*)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::_M_invoke(std::_Any_data const&, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode&&, adios2::helper::Comm&&) (__functor=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:291
#27 0x00007f73cfa05d30 in std::function<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::operator()(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm) const (this=0x76b548, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:622
#28 0x00007f73cf9ca4cb in adios2::core::IO::Open (this=0x9fc1f0, name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.cpp:643
#29 0x00007f73cf9cab12 in adios2::core::IO::Open (this=0x9fc1f0, name=, mode=adios2::Mode::Write) at ../ADIOS2/source/adios2/core/IO.cpp:665
#30 0x00007f73d0742fc6 in adios2::IO::Open (this=0x7ffe6eebeb90, name=, mode=adios2::Mode::Write) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#31 0x00000000004258d8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x9bc530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:65
#32 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x9bc530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#33 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x9bc530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#34 0x000000000044ecc9 in testing::Test::Run (this=0x9bc530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#35 0x000000000044f869 in testing::TestInfo::Run (this=0x9bc120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#36 0x00000000004500fe in testing::TestSuite::Run (this=0x9b6960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#37 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x76e300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#38 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x76e300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#39 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x76e300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#40 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#41 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#42 0x0000000000428c52 in main (argc=2, argv=0x7ffe6eebf408) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14001) detached]
[New LWP 14030]
[New LWP 14035]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f62f4888a09 in poll () from /lib64/libc.so.6
#0  0x00007f62f4888a09 in poll () from /lib64/libc.so.6
#1  0x00007f62f42bd57d in poll_dispatch (base=0x14ae0a0, tv=<optimized out>) at poll.c:182
#2  0x00007f62f42b4a65 in event_base_loop (base=0x14ae0a0, flags=2) at event.c:1992
#3  0x00007f62f458bd96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f62f458beeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f62f45922e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f62f5cf7919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f62f5cf085d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f62f5d2bca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f62f63756a3 in adios2::helper::CommImplMPI::Duplicate (this=0x16f7000, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007f62f55242d6 in adios2::helper::Comm::Duplicate (this=0x16f6d88, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007f62f5446af9 in adios2::core::IO::Open (this=0x16fbc30, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007f62f61befc6 in adios2::IO::Open (this=0x7ffeedc39b78, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x16b8530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x16b8530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x16b8530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x16b8530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x16b8120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x16b2960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x146a300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x146a300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x146a300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7ffeedc3a788) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14002) detached]
[New LWP 14033]
[New LWP 14037]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f3466c0da09 in poll () from /lib64/libc.so.6
#0  0x00007f3466c0da09 in poll () from /lib64/libc.so.6
#1  0x00007f346664257d in poll_dispatch (base=0x166d0a0, tv=<optimized out>) at poll.c:182
#2  0x00007f3466639a65 in event_base_loop (base=0x166d0a0, flags=2) at event.c:1992
#3  0x00007f3466910d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f3466910eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f34669172e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f346807c919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f346807585d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f34680b0ca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f34686fa6a3 in adios2::helper::CommImplMPI::Duplicate (this=0x18a6cd0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007f34678a92d6 in adios2::helper::Comm::Duplicate (this=0x18a6a18, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007f34677cbaf9 in adios2::core::IO::Open (this=0x18bf4a0, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007f3468543fc6 in adios2::IO::Open (this=0x7ffddfc4e138, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x18774b0) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x18774b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x18774b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x18774b0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x18770a0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x186b370) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x1629300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1629300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1629300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7ffddfc4ed38) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14003) detached]
[New LWP 14012]
[New LWP 14014]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f7568d33a09 in poll () from /lib64/libc.so.6
#0  0x00007f7568d33a09 in poll () from /lib64/libc.so.6
#1  0x00007f756876857d in poll_dispatch (base=0x59e0a0, tv=<optimized out>) at poll.c:182
#2  0x00007f756875fa65 in event_base_loop (base=0x59e0a0, flags=2) at event.c:1992
#3  0x00007f7568a36d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f7568a36eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f7568a3d2e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f756a1a2919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f756a19b85d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f756a1d6ca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f756a8206a3 in adios2::helper::CommImplMPI::Duplicate (this=0x7d7d00, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007f75699cf2d6 in adios2::helper::Comm::Duplicate (this=0x7d7a88, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007f75698f1af9 in adios2::core::IO::Open (this=0x7dc920, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007f756a669fc6 in adios2::IO::Open (this=0x7ffe82aaba78, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x7a8460) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x7a8460, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x7a8460, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x7a8460) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x7a8050) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x79c3b0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x55a300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55a300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55a300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7ffe82aac678) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14004) detached]
[New LWP 14016]
[New LWP 14019]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fecac66ba09 in poll () from /lib64/libc.so.6
#0  0x00007fecac66ba09 in poll () from /lib64/libc.so.6
#1  0x00007fecac0a057d in poll_dispatch (base=0x1a9b0a0, tv=<optimized out>) at poll.c:182
#2  0x00007fecac097a65 in event_base_loop (base=0x1a9b0a0, flags=2) at event.c:1992
#3  0x00007fecac36ed96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007fecac36eeeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007fecac3752e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007fecadada919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007fecadad385d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007fecadb0eca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007fecae1586a3 in adios2::helper::CommImplMPI::Duplicate (this=0x1cc5be0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007fecad3072d6 in adios2::helper::Comm::Duplicate (this=0x1cc5928, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007fecad229af9 in adios2::core::IO::Open (this=0x1cde360, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007fecadfa1fc6 in adios2::IO::Open (this=0x7ffeb9f1bb58, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x1ca5530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x1ca5530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x1ca5530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x1ca5530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x1ca5120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x1c9f960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x1a57300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1a57300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1a57300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7ffeb9f1c768) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14005) detached]
[New LWP 14017]
[New LWP 14020]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f99c5605a09 in poll () from /lib64/libc.so.6
#0  0x00007f99c5605a09 in poll () from /lib64/libc.so.6
#1  0x00007f99c503a57d in poll_dispatch (base=0x17050a0, tv=<optimized out>) at poll.c:182
#2  0x00007f99c5031a65 in event_base_loop (base=0x17050a0, flags=2) at event.c:1992
#3  0x00007f99c5308d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f99c5308eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f99c530f2e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f99c6c2a100 in mca_pml_ob1_recv () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f99c6af8df6 in ompi_coll_base_gather_intra_binomial () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f99c6b0d511 in ompi_coll_tuned_gather_intra_dec_fixed () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f99c6ab5542 in PMPI_Gather () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#10 0x00007f99c70f30fa in adios2::helper::CommImplMPI::Gather (this=0x1950ac0, sendbuf=0x7ffc84f2e010, sendcount=1, sendtype=adios2::helper::CommImpl::Datatype::UnsignedLong, recvbuf=0x0, recvcount=1, recvtype=adios2::helper::CommImpl::Datatype::UnsignedLong, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:336
#11 0x00007f99c6137905 in adios2::helper::Comm::Gather<unsigned long, unsigned long> (this=0x1951f18, sendbuf=0x7ffc84f2e010, sendcount=1, recvbuf=0x0, recvcount=1, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.inl:211
#12 0x00007f99c6374a14 in adios2::helper::Comm::GatherArrays<unsigned long> (this=0x1951f18, source=0x7ffc84f2e010, sourceCount=1, destination=0x0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:30
#13 0x00007f99c63742a3 in adios2::helper::Comm::GatherValues<unsigned long> (this=0x1951f18, source=296, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:48
#14 0x00007f99c642fece in adios2::helper::Comm::GathervVectors<char> (this=0x1951f18, in=std::vector of length 296, capacity 296 = {...}, out=std::vector of length 0, capacity 0, position=@0x1952100: 0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:83
#15 0x00007f99c641526a in adios2::format::BP3Serializer::AggregateCollectiveMetadataIndices (this=0x1951f70, comm=..., bufferSTL=...) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:544
#16 0x00007f99c6414ab2 in adios2::format::BP3Serializer::AggregateCollectiveMetadata (this=0x1951f70, comm=..., bufferSTL=..., inMetadataBuffer=true) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:196
#17 0x00007f99c62eca4a in adios2::core::engine::BP3Writer::WriteCollectiveMetadataFile (this=0x1951ec0, isFinal=false) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:329
#18 0x00007f99c62ec8d0 in adios2::core::engine::BP3Writer::Flush (this=0x1951ec0, transportIndex=-1) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:119
#19 0x00007f99c62ec7c3 in adios2::core::engine::BP3Writer::EndStep (this=0x1951ec0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:107
#20 0x00007f99c6ed6a9b in adios2::Engine::EndStep (this=0x7ffc84f2f4e0) at ../ADIOS2/bindings/CXX11/adios2/cxx11/Engine.cpp:113
#21 0x00000000004266f8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x190f530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:109
#22 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x190f530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#23 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x190f530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#24 0x000000000044ecc9 in testing::Test::Run (this=0x190f530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#25 0x000000000044f869 in testing::TestInfo::Run (this=0x190f120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#26 0x00000000004500fe in testing::TestSuite::Run (this=0x1909960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#27 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x16c1300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#28 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x16c1300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#29 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x16c1300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#30 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#31 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#32 0x0000000000428c52 in main (argc=2, argv=0x7ffc84f2fd98) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14006) detached]
[New LWP 14018]
[New LWP 14021]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f04e9b9ea09 in poll () from /lib64/libc.so.6
#0  0x00007f04e9b9ea09 in poll () from /lib64/libc.so.6
#1  0x00007f04e95d357d in poll_dispatch (base=0x8590a0, tv=<optimized out>) at poll.c:182
#2  0x00007f04e95caa65 in event_base_loop (base=0x8590a0, flags=2) at event.c:1992
#3  0x00007f04e98a1d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f04e98a1eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f04e98a82e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f04eb02630c in ompi_request_default_wait () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f04eb09872f in ompi_coll_base_barrier_intra_recursivedoubling () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f04eb03df98 in PMPI_Barrier () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f04eb68ba5d in adios2::helper::CommImplMPI::Barrier (this=0xa86100, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:276
#10 0x00007f04ea83a569 in adios2::helper::Comm::Barrier (this=0xa872b8, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:75
#11 0x00007f04eabddc72 in adios2::transportman::TransportMan::MkDirsBarrier (this=0xa877f0, fileNames=std::vector of length 32, capacity 32, parametersVector=std::vector of length 1, capacity 1 = {...}, nodeLocal=false) at ../ADIOS2/source/adios2/toolkit/transportman/TransportMan.cpp:104
#12 0x00007f04ea88616a in adios2::core::engine::BP3Writer::InitTransports (this=0xa87260) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:195
#13 0x00007f04ea88431a in adios2::core::engine::BP3Writer::Init (this=0xa87260) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:133
#14 0x00007f04ea88417f in adios2::core::engine::BP3Writer::BP3Writer (this=0xa87260, io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:34
#15 0x00007f04ea79dfc7 in __gnu_cxx::new_allocator<adios2::core::engine::BP3Writer>::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe2b60c4c0, __p=0xa87260, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/ext/new_allocator.h:156
#16 0x00007f04ea79dd2c in std::allocator_traits<std::allocator<adios2::core::engine::BP3Writer> >::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __p=0xa87260, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/alloc_traits.h:512
#17 0x00007f04ea79da60 in std::_Sp_counted_ptr_inplace<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0xa87250, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:551
#18 0x00007f04ea79d7f5 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe2b60c748, __p=@0x7ffe2b60c740: 0x0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:683
#19 0x00007f04ea79d6d6 in std::__shared_ptr<adios2::core::engine::BP3Writer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe2b60c740, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:1376
#20 0x00007f04ea79d62c in std::shared_ptr<adios2::core::engine::BP3Writer>::shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7ffe2b60c740, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:408
#21 0x00007f04ea79d565 in std::allocate_shared<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:861
#22 0x00007f04ea79d443 in std::make_shared<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:877
#23 0x00007f04ea7836e5 in adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer> (io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.h:473
#24 0x00007f04eb689fc4 in std::__invoke_impl<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__f=@0x812548: 0x7f04ea783690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:60
#25 0x00007f04eb689eca in std::__invoke_r<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__fn=@0x812548: 0x7f04ea783690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:141
#26 0x00007f04eb689d4a in std::_Function_handler<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), std::shared_ptr<adios2::core::Engine> (*)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::_M_invoke(std::_Any_data const&, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode&&, adios2::helper::Comm&&) (__functor=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:291
#27 0x00007f04ea797d30 in std::function<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::operator()(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm) const (this=0x812548, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:622
#28 0x00007f04ea75c4cb in adios2::core::IO::Open (this=0xa84ee0, name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.cpp:643
#29 0x00007f04ea75cb12 in adios2::core::IO::Open (this=0xa84ee0, name=, mode=adios2::Mode::Write) at ../ADIOS2/source/adios2/core/IO.cpp:665
#30 0x00007f04eb4d4fc6 in adios2::IO::Open (this=0x7ffe2b60df70, name=, mode=adios2::Mode::Write) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#31 0x00000000004258d8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0xa63530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:65
#32 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xa63530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#33 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xa63530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#34 0x000000000044ecc9 in testing::Test::Run (this=0xa63530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#35 0x000000000044f869 in testing::TestInfo::Run (this=0xa63120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#36 0x00000000004500fe in testing::TestSuite::Run (this=0xa5d960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#37 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x815300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#38 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x815300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#39 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x815300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#40 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#41 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#42 0x0000000000428c52 in main (argc=2, argv=0x7ffe2b60e7d8) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14007) detached]
[New LWP 14023]
[New LWP 14026]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fedfd101a09 in poll () from /lib64/libc.so.6
#0  0x00007fedfd101a09 in poll () from /lib64/libc.so.6
#1  0x00007fedfcb3657d in poll_dispatch (base=0x8380a0, tv=<optimized out>) at poll.c:182
#2  0x00007fedfcb2da65 in event_base_loop (base=0x8380a0, flags=2) at event.c:1992
#3  0x00007fedfce04d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007fedfce04eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007fedfce0b2e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007fedfe726100 in mca_pml_ob1_recv () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007fedfe5f4df6 in ompi_coll_base_gather_intra_binomial () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007fedfe609511 in ompi_coll_tuned_gather_intra_dec_fixed () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007fedfe5b1542 in PMPI_Gather () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#10 0x00007fedfebef0fa in adios2::helper::CommImplMPI::Gather (this=0xa744f0, sendbuf=0x7fff8399c390, sendcount=1, sendtype=adios2::helper::CommImpl::Datatype::UnsignedLong, recvbuf=0x0, recvcount=1, recvtype=adios2::helper::CommImpl::Datatype::UnsignedLong, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:336
#11 0x00007fedfdc33905 in adios2::helper::Comm::Gather<unsigned long, unsigned long> (this=0xa75878, sendbuf=0x7fff8399c390, sendcount=1, recvbuf=0x0, recvcount=1, root=0, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.inl:211
#12 0x00007fedfde70a14 in adios2::helper::Comm::GatherArrays<unsigned long> (this=0xa75878, source=0x7fff8399c390, sourceCount=1, destination=0x0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:30
#13 0x00007fedfde702a3 in adios2::helper::Comm::GatherValues<unsigned long> (this=0xa75878, source=296, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:48
#14 0x00007fedfdf2bece in adios2::helper::Comm::GathervVectors<char> (this=0xa75878, in=std::vector of length 296, capacity 296 = {...}, out=std::vector of length 0, capacity 0, position=@0xa75a60: 0, rankDestination=0) at ../ADIOS2/source/adios2/helper/adiosComm.inl:83
#15 0x00007fedfdf1126a in adios2::format::BP3Serializer::AggregateCollectiveMetadataIndices (this=0xa758d0, comm=..., bufferSTL=...) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:544
#16 0x00007fedfdf10ab2 in adios2::format::BP3Serializer::AggregateCollectiveMetadata (this=0xa758d0, comm=..., bufferSTL=..., inMetadataBuffer=true) at ../ADIOS2/source/adios2/toolkit/format/bp/bp3/BP3Serializer.cpp:196
#17 0x00007fedfdde8a4a in adios2::core::engine::BP3Writer::WriteCollectiveMetadataFile (this=0xa75820, isFinal=false) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:329
#18 0x00007fedfdde88d0 in adios2::core::engine::BP3Writer::Flush (this=0xa75820, transportIndex=-1) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:119
#19 0x00007fedfdde87c3 in adios2::core::engine::BP3Writer::EndStep (this=0xa75820) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:107
#20 0x00007fedfe9d2a9b in adios2::Engine::EndStep (this=0x7fff8399d860) at ../ADIOS2/bindings/CXX11/adios2/cxx11/Engine.cpp:113
#21 0x00000000004266f8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0xa42530) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:109
#22 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xa42530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#23 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xa42530, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#24 0x000000000044ecc9 in testing::Test::Run (this=0xa42530) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#25 0x000000000044f869 in testing::TestInfo::Run (this=0xa42120) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#26 0x00000000004500fe in testing::TestSuite::Run (this=0xa3c960) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#27 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x7f4300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#28 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x7f4300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#29 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x7f4300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#30 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#31 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#32 0x0000000000428c52 in main (argc=2, argv=0x7fff8399e128) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14008) detached]
[New LWP 14022]
[New LWP 14024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f07a42aca09 in poll () from /lib64/libc.so.6
#0  0x00007f07a42aca09 in poll () from /lib64/libc.so.6
#1  0x00007f07a3ce157d in poll_dispatch (base=0x8320a0, tv=<optimized out>) at poll.c:182
#2  0x00007f07a3cd8a65 in event_base_loop (base=0x8320a0, flags=2) at event.c:1992
#3  0x00007f07a3fafd96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f07a3fafeeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f07a3fb62e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f07a573430c in ompi_request_default_wait () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f07a57a672f in ompi_coll_base_barrier_intra_recursivedoubling () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f07a574bf98 in PMPI_Barrier () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f07a5d99a5d in adios2::helper::CommImplMPI::Barrier (this=0xa5f080, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:276
#10 0x00007f07a4f48569 in adios2::helper::Comm::Barrier (this=0xa60238, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:75
#11 0x00007f07a52ebc72 in adios2::transportman::TransportMan::MkDirsBarrier (this=0xa60770, fileNames=std::vector of length 32, capacity 32, parametersVector=std::vector of length 1, capacity 1 = {...}, nodeLocal=false) at ../ADIOS2/source/adios2/toolkit/transportman/TransportMan.cpp:104
#12 0x00007f07a4f9416a in adios2::core::engine::BP3Writer::InitTransports (this=0xa601e0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:195
#13 0x00007f07a4f9231a in adios2::core::engine::BP3Writer::Init (this=0xa601e0) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:133
#14 0x00007f07a4f9217f in adios2::core::engine::BP3Writer::BP3Writer (this=0xa601e0, io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/engine/bp3/BP3Writer.cpp:34
#15 0x00007f07a4eabfc7 in __gnu_cxx::new_allocator<adios2::core::engine::BP3Writer>::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7fff5998d360, __p=0xa601e0, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/ext/new_allocator.h:156
#16 0x00007f07a4eabd2c in std::allocator_traits<std::allocator<adios2::core::engine::BP3Writer> >::construct<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __p=0xa601e0, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/alloc_traits.h:512
#17 0x00007f07a4eaba60 in std::_Sp_counted_ptr_inplace<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0xa601d0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:551
#18 0x00007f07a4eab7f5 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7fff5998d5e8, __p=@0x7fff5998d5e0: 0x0, __a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:683
#19 0x00007f07a4eab6d6 in std::__shared_ptr<adios2::core::engine::BP3Writer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7fff5998d5e0, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr_base.h:1376
#20 0x00007f07a4eab62c in std::shared_ptr<adios2::core::engine::BP3Writer>::shared_ptr<std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (this=0x7fff5998d5e0, __tag=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:408
#21 0x00007f07a4eab565 in std::allocate_shared<adios2::core::engine::BP3Writer, std::allocator<adios2::core::engine::BP3Writer>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__a=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:861
#22 0x00007f07a4eab443 in std::make_shared<adios2::core::engine::BP3Writer, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, adios2::helper::Comm> (__args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/shared_ptr.h:877
#23 0x00007f07a4e916e5 in adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer> (io=..., name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.h:473
#24 0x00007f07a5d97fc4 in std::__invoke_impl<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__f=@0x7eb548: 0x7f07a4e91690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:60
#25 0x00007f07a5d97eca in std::__invoke_r<std::shared_ptr<adios2::core::Engine>, std::shared_ptr<adios2::core::Engine> (*&)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm> (__fn=@0x7eb548: 0x7f07a4e91690 <adios2::core::IO::MakeEngine<adios2::core::engine::BP3Writer>(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/invoke.h:141
#26 0x00007f07a5d97d4a in std::_Function_handler<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm), std::shared_ptr<adios2::core::Engine> (*)(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::_M_invoke(std::_Any_data const&, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode&&, adios2::helper::Comm&&) (__functor=..., __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:291
#27 0x00007f07a4ea5d30 in std::function<std::shared_ptr<adios2::core::Engine> (adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm)>::operator()(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, adios2::helper::Comm) const (this=0x7eb548, __args=..., __args=..., __args=..., __args=...) at /usr/bin/../lib64/gcc/x86_64-suse-linux/10/../../../../include/c++/10/bits/std_function.h:622
#28 0x00007f07a4e6a4cb in adios2::core::IO::Open (this=0xa5de60, name=, mode=adios2::Mode::Write, comm=...) at ../ADIOS2/source/adios2/core/IO.cpp:643
#29 0x00007f07a4e6ab12 in adios2::core::IO::Open (this=0xa5de60, name=, mode=adios2::Mode::Write) at ../ADIOS2/source/adios2/core/IO.cpp:665
#30 0x00007f07a5be2fc6 in adios2::IO::Open (this=0x7fff5998ee10, name=, mode=adios2::Mode::Write) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#31 0x00000000004258d8 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0xa3c4b0) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:65
#32 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xa3c4b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#33 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xa3c4b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#34 0x000000000044ecc9 in testing::Test::Run (this=0xa3c4b0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#35 0x000000000044f869 in testing::TestInfo::Run (this=0xa3c0a0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#36 0x00000000004500fe in testing::TestSuite::Run (this=0xa30370) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#37 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x7ee300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#38 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x7ee300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#39 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x7ee300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#40 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#41 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#42 0x0000000000428c52 in main (argc=2, argv=0x7fff5998f688) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14009) detached]
[New LWP 14025]
[New LWP 14027]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007febda350a09 in poll () from /lib64/libc.so.6
#0  0x00007febda350a09 in poll () from /lib64/libc.so.6
#1  0x00007febd9d8557d in poll_dispatch (base=0xe950a0, tv=<optimized out>) at poll.c:182
#2  0x00007febd9d7ca65 in event_base_loop (base=0xe950a0, flags=2) at event.c:1992
#3  0x00007febda053d96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007febda053eeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007febda05a2e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007febdb7bf919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007febdb7b885d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007febdb7f3ca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007febdbe3d6a3 in adios2::helper::CommImplMPI::Duplicate (this=0x10ced50, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007febdafec2d6 in adios2::helper::Comm::Duplicate (this=0x10cead8, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007febdaf0eaf9 in adios2::core::IO::Open (this=0x10d3b30, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007febdbc86fc6 in adios2::IO::Open (this=0x7fff9616d8d8, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x109f4b0) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x109f4b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x109f4b0, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x109f4b0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x109f0a0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x1093370) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0xe51300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xe51300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xe51300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7fff9616e4d8) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14010) detached]
[New LWP 14038]
[New LWP 14039]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f0c6c157a09 in poll () from /lib64/libc.so.6
#0  0x00007f0c6c157a09 in poll () from /lib64/libc.so.6
#1  0x00007f0c6bb8c57d in poll_dispatch (base=0x14aa0a0, tv=<optimized out>) at poll.c:182
#2  0x00007f0c6bb83a65 in event_base_loop (base=0x14aa0a0, flags=2) at event.c:1992
#3  0x00007f0c6be5ad96 in opal_progress_events.isra () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#4  0x00007f0c6be5aeeb in opal_progress () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#5  0x00007f0c6be612e5 in ompi_sync_wait_mt () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libopen-pal.so.40
#6  0x00007f0c6d5c6919 in ompi_comm_nextcid () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#7  0x00007f0c6d5bf85d in ompi_comm_dup_with_info () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#8  0x00007f0c6d5faca0 in PMPI_Comm_dup () from /opt/spack/opt/spack/linux-opensuse15-skylake/gcc-10.4.0/openmpi-4.1.5-gcl3bpuekasgri2vcrjrp7kmpkehnmbh/lib64/libmpi.so.40
#9  0x00007f0c6dc446a3 in adios2::helper::CommImplMPI::Duplicate (this=0x16d4b10, hint=) at ../ADIOS2/source/adios2/helper/adiosCommMPI.cpp:233
#10 0x00007f0c6cdf32d6 in adios2::helper::Comm::Duplicate (this=0x16d4858, hint=) at ../ADIOS2/source/adios2/helper/adiosComm.cpp:58
#11 0x00007f0c6cd15af9 in adios2::core::IO::Open (this=0x16d9530, name=, mode=adios2::Mode::Read) at ../ADIOS2/source/adios2/core/IO.cpp:665
#12 0x00007f0c6da8dfc6 in adios2::IO::Open (this=0x7ffcfaddfd38, name=, mode=adios2::Mode::Read) at ../ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:109
#13 0x0000000000426893 in BPChangingShapeWithinStep_MultiBlock_Test::TestBody (this=0x16b4460) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:122
#14 0x0000000000488d94 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x16b4460, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#15 0x000000000046d67b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x16b4460, method=&virtual testing::Test::TestBody(), location=0x490e2f "the test body") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#16 0x000000000044ecc9 in testing::Test::Run (this=0x16b4460) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2670
#17 0x000000000044f869 in testing::TestInfo::Run (this=0x16b4050) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2849
#18 0x00000000004500fe in testing::TestSuite::Run (this=0x16a83b0) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:3008
#19 0x0000000000460716 in testing::internal::UnitTestImpl::RunAllTests (this=0x1466300) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5866
#20 0x000000000048b9f4 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1466300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2595
#21 0x000000000046fbab in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1466300, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4602a0 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x491708 "auxiliary test code (environments or event listeners)") at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:2631
#22 0x0000000000460251 in testing::UnitTest::Run (this=0x4c4ee8 <testing::UnitTest::GetInstance()::instance>) at ../ADIOS2/thirdparty/GTest/googletest/googletest/src/gtest.cc:5440
#23 0x000000000042a531 in RUN_ALL_TESTS () at ../ADIOS2/thirdparty/GTest/googletest/googletest/include/gtest/gtest.h:2284
#24 0x0000000000428c52 in main (argc=2, argv=0x7ffcfade0938) at ../ADIOS2/testing/adios2/engine/bp/TestBPChangingShapeWithinStep.cpp:272
[Inferior 1 (process 14011) detached]

vicentebolea · 2023-11-14T21:46:30Z

I found that this problem occurs on my laptop when connected to a ethernet network but not when connected to the wifi network. I could figure out that this is due to openmpi picking certain network interfaces, a easy workaround is to use the lo interface

eisenhauer · 2023-11-14T22:56:06Z

I found that this problem occurs on my laptop when connected to a ethernet network but not when connected to the wifi network. I could figure out that this is due to openmpi picking certain network interfaces, a easy workaround is to use the lo interface

Well, maybe... But why only this test? MGard doesn't use MPI, but it does use OpenMP. I build MGard manually on my OSX laptop and these tests have no problems. I'm wondering if on whatever platform this is running OpenMP isn't playing well with MPI, resulting in the issues that you're seeing. (I could use the docker image and dig further, but I'm off at a conference, so little spare time for the moment.)

vicentebolea · 2023-11-15T00:25:09Z

Well, maybe... But why only this test? MGard doesn't use MPI, but it does use OpenMP. I build MGard manually on my OSX laptop and these tests have no problems. I'm wondering if on whatever platform this is running OpenMP isn't playing well with MPI, resulting in the issues that you're seeing. (I could use the docker image and dig further, but I'm off at a conference, so little spare time for the moment.)

hmm, so two things happened here:

The real culcript was the that we were using n4) number of mpi tasks and each of the task was using n2 number of threads (n=2) this was just too much for the weak VPS that github gives us.
The interface thing was affecting time to time different mpi test in random fashion

vicentebolea · 2023-11-15T02:23:13Z

@scottwittenburg if this makes sense to you please approve so that it gets merged and we get a green dashboard 😎

vicentebolea · 2023-11-15T03:35:34Z

Yay, it's all green again :)

eisenhauer

Looks good.

scottwittenburg · 2023-11-15T18:12:50Z

I've rebased on this change, and now my ci reproducer builds are happening in serial, which seems like an unintended consequence.

vicentebolea force-pushed the increase-timeout-mgard-tests branch 2 times, most recently from f920f4e to 5401f7c Compare November 8, 2023 18:05

vicentebolea force-pushed the increase-timeout-mgard-tests branch from 5401f7c to 653606e Compare November 9, 2023 00:02

vicentebolea force-pushed the increase-timeout-mgard-tests branch from e6b61f5 to eba481e Compare November 13, 2023 18:34

vicentebolea mentioned this pull request Nov 14, 2023

ci: use mpich built with ch3:sock to speed up tests #3883

Merged

vicentebolea force-pushed the increase-timeout-mgard-tests branch 2 times, most recently from 61e2022 to 452e541 Compare November 14, 2023 21:43

vicentebolea force-pushed the increase-timeout-mgard-tests branch from e6f9790 to 77729e7 Compare November 14, 2023 23:48

vicentebolea force-pushed the increase-timeout-mgard-tests branch from 24f4610 to 4cbf92e Compare November 15, 2023 00:33

vicentebolea changed the title ~~ci: increase timeout BPChangingShapeWithinStep.MultiBlock~~ ci: Set openmpi and openmp parameters to make an efficient use of the CI Nov 15, 2023

vicentebolea self-assigned this Nov 15, 2023

vicentebolea requested a review from scottwittenburg November 15, 2023 01:28

vicentebolea force-pushed the increase-timeout-mgard-tests branch from 4cbf92e to 954ecca Compare November 15, 2023 02:13

vicentebolea enabled auto-merge November 15, 2023 02:22

ci: set openmpi and openmp params

92ebfed

vicentebolea force-pushed the increase-timeout-mgard-tests branch from 954ecca to 92ebfed Compare November 15, 2023 02:23

eisenhauer approved these changes Nov 15, 2023

View reviewed changes

vicentebolea merged commit 3c21c37 into ornladios:master Nov 15, 2023
33 checks passed

vicentebolea deleted the increase-timeout-mgard-tests branch November 15, 2023 19:16

BrewTestBot mentioned this pull request Apr 4, 2024

adios2 2.10.0 Homebrew/homebrew-core#167957

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Set openmpi and openmp parameters to make an efficient use of the CI #3908

ci: Set openmpi and openmp parameters to make an efficient use of the CI #3908

vicentebolea commented Nov 8, 2023

anagainaru commented Nov 8, 2023

eisenhauer commented Nov 8, 2023

vicentebolea commented Nov 13, 2023 •

edited

Loading

vicentebolea commented Nov 14, 2023

eisenhauer commented Nov 14, 2023

vicentebolea commented Nov 15, 2023

vicentebolea commented Nov 15, 2023

vicentebolea commented Nov 15, 2023

eisenhauer left a comment

scottwittenburg commented Nov 15, 2023

ci: Set openmpi and openmp parameters to make an efficient use of the CI #3908

ci: Set openmpi and openmp parameters to make an efficient use of the CI #3908

Conversation

vicentebolea commented Nov 8, 2023

anagainaru commented Nov 8, 2023

eisenhauer commented Nov 8, 2023

vicentebolea commented Nov 13, 2023 • edited Loading

vicentebolea commented Nov 14, 2023

eisenhauer commented Nov 14, 2023

vicentebolea commented Nov 15, 2023

vicentebolea commented Nov 15, 2023

vicentebolea commented Nov 15, 2023

eisenhauer left a comment

Choose a reason for hiding this comment

scottwittenburg commented Nov 15, 2023

vicentebolea commented Nov 13, 2023 •

edited

Loading