aarch64 test failed after build with gcc and hmpi

[aiye123]

The title of the issue should start with Defect: followed by a
succinct title.

Please make sure to put any logs, terminal output, or code in
fenced code blocks. Please also read the contributing guidelines
before submitting a new issue.

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information.

• I am reporting a bug others will be able to reproduce and not asking a question or requesting a new feature.

System information including:

• OpenCoarrays Version: 2.10.1
• Fortran Compiler: gfortran
• C compiler used for building lib: gcc
• Installation method: install.sh(cmake)
• All flags & options passed to the installer ${JARVIS_TMP}/OpenCoarrays-2.10.1/ -DCMAKE_INSTALL_PREFIX=$1
  -DCMAKE_BUILD_TYPE:STRING="Debug"
• Output of uname -a: Linux host-10-208-134-176 4.19.90-2012.4.0.0053.oe1.aarch64 tests dis_transpose: test passed  #1 SMP Mon Dec 21 14:33:58 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
• MPI library being used: hmpi1.1.1(openmpi 4)
• Machine architecture and number of physical cores:arm64 16cores
• Version of CMake: 3.16.5

To help us debug your issue please explain:

What you were trying to do (and why)

test opencoarrays on openEuler system

What happened (include command output, screenshots, logs, etc.)

run test after build
The following tests FAILED:
14 - alloc_comp_get_convert_nums (Failed)
29 - alloc_comp_send_convert_nums (Failed)
Errors while running CTest

What you expected to happen

all test should pass

Step-by-step reproduction instructions to reproduce the error/bug

rm -rf OpenCoarrays-2.10.1
tar -xvf ${JARVIS_DOWNLOAD}/OpenCoarrays-2.10.1.tar.gz
cd OpenCoarrays-2.10.1
rm -rf opencoarrays-build
mkdir opencoarrays-build && cd opencoarrays-build
cmake ${JARVIS_TMP}/OpenCoarrays-2.10.1/ -DCMAKE_INSTALL_PREFIX=$1 \
-DCMAKE_BUILD_TYPE:STRING="Debug"
make -j
make install -j

export OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 && ctest --output-on-failure --schedule-random --repeat-until-fail 1 --timeout 200

[aiye123]

10/81 Test #29: alloc_comp_send_convert_nums .....................***Failed  Required regular expression not found. Regex=[Test passed.
]  0.10 sec
2/2: _gfortran_caf_register(1191) size = 448, type = 0, token = 0x442310, desc = 0xffffdeca03a0
1/2: _gfortran_caf_init(950) The mpi memory model is: unified  (0x0, 1).
1/2: _gfortran_caf_register(1191) size = 448, type = 0, token = 0x442310, desc = 0xffffdc5ab870
2/2: _gfortran_caf_register(1307) Token 0x2b90ea50 on exit: mpi_caf_token_t { (local_)memptr: 0xffff89cd02c8 (size: 448), memptr_win: 730917488 }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdeca0380, desc = 0xffffdeca0120
1/2: _gfortran_caf_register(1307) Token 0x27cce960 on exit: mpi_caf_token_t { (local_)memptr: 0xfffd7ce30108 (size: 448), memptr_win: 667740544 }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdc5ab850, desc = 0xffffdc5ab5f0
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b9176f0 (size: 16, mpi-address: 0x2b9176f0) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b9176f0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdeca0388, desc = 0xffffdeca0150
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd7790 (size: 16, mpi-address: 0x27cd7790) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7790 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdc5ab858, desc = 0xffffdc5ab620
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917730 (size: 16, mpi-address: 0x2b917730) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917730 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdeca0390, desc = 0xffffdeca0180
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917770 (size: 16, mpi-address: 0x2b917770) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917770 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdeca0398, desc = 0xffffdeca01b0
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917810 (size: 16, mpi-address: 0x2b917810) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917810 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdeca0258, desc = 0xffffdeca0200
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd7830 (size: 16, mpi-address: 0x27cd7830) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7830 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdc5ab860, desc = 0xffffdc5ab650
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917d70 (size: 16, mpi-address: 0x2b917d70) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917d70 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdeca02b8, desc = 0xffffdeca0260
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917e10 (size: 16, mpi-address: 0x2b917e10) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917e10 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdeca0318, desc = 0xffffdeca02c0
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd78d0 (size: 16, mpi-address: 0x27cd78d0) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd78d0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffdc5ab868, desc = 0xffffdc5ab680
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd7970 (size: 16, mpi-address: 0x27cd7970) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7970 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdc5ab728, desc = 0xffffdc5ab6d0
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917eb0 (size: 16, mpi-address: 0x2b917eb0) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917eb0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdeca0378, desc = 0xffffdeca0320
2/2: _gfortran_caf_register(1216) Attach slave token 0x2b917f50 (size: 16, mpi-address: 0x2b917f50) to global_dynamic_window = 730824144
2/2: _gfortran_caf_register(1257) Slave token 0x2b917f50 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd7ed0 (size: 16, mpi-address: 0x27cd7ed0) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7ed0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdc5ab788, desc = 0xffffdc5ab730
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd7f70 (size: 16, mpi-address: 0x27cd7f70) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7f70 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdc5ab7e8, desc = 0xffffdc5ab790
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd8010 (size: 16, mpi-address: 0x27cd8010) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd8010 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 1, type = 8, token = 0xffff89cd0468, desc = 0xffffdeca00e0
2/2: _gfortran_caf_register(1240) Attach mem 0x2b9184b0 (mpi-address: 0x2b9184b0) to global_dynamic_window = 730824144 on slave_token 0x2b9176f0, size 1, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0x2b9176f0 on exit: mpi_caf_slave_token_t { memptr: 0x2b9184b0, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffdc5ab848, desc = 0xffffdc5ab7f0
1/2: _gfortran_caf_register(1216) Attach slave token 0x27cd80b0 (size: 16, mpi-address: 0x27cd80b0) to global_dynamic_window = 667647248
1/2: _gfortran_caf_register(1257) Slave token 0x27cd80b0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xffff89cd0470, desc = 0xffffdeca00e0
2/2: _gfortran_caf_register(1240) Attach mem 0x2b918530 (mpi-address: 0x2b918530) to global_dynamic_window = 730824144 on slave_token 0x2b917730, size 4, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0x2b917730 on exit: mpi_caf_slave_token_t { memptr: 0x2b918530, desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 5, type = 8, token = 0xffff89cd0340, desc = 0xffff89cd02e8
2/2: _gfortran_caf_register(1240) Attach mem 0x2b9185b0 (mpi-address: 0x2b9185b0) to global_dynamic_window = 730824144 on slave_token 0x2b917d70, size 5, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff89cd02e8 (mpi-address: 281472993657576) to global_dynamic_window 730824144 at address 0x2b917d78, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0x2b917d70 on exit: mpi_caf_slave_token_t { memptr: 0x2b9185b0, desc: 0xffff89cd02e8 }
2/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xffff89cd03a0, desc = 0xffff89cd0348
1/2: _gfortran_caf_register(1191) size = 1, type = 8, token = 0xfffd7ce302a8, desc = 0xffffdc5ab5b0
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8610 (mpi-address: 0x27cd8610) to global_dynamic_window = 667647248 on slave_token 0x27cd7790, size 1, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7790 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8610, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xfffd7ce302b0, desc = 0xffffdc5ab5b0
2/2: _gfortran_caf_register(1240) Attach mem 0x2b918630 (mpi-address: 0x2b918630) to global_dynamic_window = 730824144 on slave_token 0x2b917e10, size 20, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff89cd0348 (mpi-address: 281472993657672) to global_dynamic_window 730824144 at address 0x2b917e18, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0x2b917e10 on exit: mpi_caf_slave_token_t { memptr: 0x2b918630, desc: 0xffff89cd0348 }
2/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xffff89cd0478, desc = 0xffffdeca00e0
2/2: _gfortran_caf_register(1240) Attach mem 0x2b9186b0 (mpi-address: 0x2b9186b0) to global_dynamic_window = 730824144 on slave_token 0x2b917770, size 4, ierr: 0
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8690 (mpi-address: 0x27cd8690) to global_dynamic_window = 667647248 on slave_token 0x27cd7830, size 4, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7830 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8690, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 5, type = 8, token = 0xfffd7ce30180, desc = 0xfffd7ce30128
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8710 (mpi-address: 0x27cd8710) to global_dynamic_window = 667647248 on slave_token 0x27cd7ed0, size 5, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffd7ce30128 (mpi-address: 281464187060520) to global_dynamic_window 667647248 at address 0x27cd7ed8, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0x2b917770 on exit: mpi_caf_slave_token_t { memptr: 0x2b9186b0, desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 8, token = 0xffff89cd0480, desc = 0xffffdeca00e0
2/2: _gfortran_caf_register(1240) Attach mem 0x2b918bf0 (mpi-address: 0x2b918bf0) to global_dynamic_window = 730824144 on slave_token 0x2b917810, size 8, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0x2b917810 on exit: mpi_caf_slave_token_t { memptr: 0x2b918bf0, desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xffff89cd0400, desc = 0xffff89cd03a8
2/2: _gfortran_caf_register(1240) Attach mem 0x2b918c70 (mpi-address: 0x2b918c70) to global_dynamic_window = 730824144 on slave_token 0x2b917eb0, size 20, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7ed0 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8710, desc: 0xfffd7ce30128 }
1/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xfffd7ce301e0, desc = 0xfffd7ce30188
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8790 (mpi-address: 0x27cd8790) to global_dynamic_window = 667647248 on slave_token 0x27cd7f70, size 20, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffd7ce30188 (mpi-address: 281464187060616) to global_dynamic_window 667647248 at address 0x27cd7f78, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7f70 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8790, desc: 0xfffd7ce30188 }
1/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xfffd7ce302b8, desc = 0xffffdc5ab5b0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff89cd03a8 (mpi-address: 281472993657768) to global_dynamic_window 730824144 at address 0x2b917eb8, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0x2b917eb0 on exit: mpi_caf_slave_token_t { memptr: 0x2b918c70, desc: 0xffff89cd03a8 }
2/2: _gfortran_caf_register(1191) size = 40, type = 8, token = 0xffff89cd0460, desc = 0xffff89cd0408
2/2: _gfortran_caf_register(1240) Attach mem 0x2b8f75d0 (mpi-address: 0x2b8f75d0) to global_dynamic_window = 730824144 on slave_token 0x2b917f50, size 40, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff89cd0408 (mpi-address: 281472993657864) to global_dynamic_window 730824144 at address 0x2b917f58, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0x2b917f50 on exit: mpi_caf_slave_token_t { memptr: 0x2b8f75d0, desc: 0xffff89cd0408 }
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8810 (mpi-address: 0x27cd8810) to global_dynamic_window = 667647248 on slave_token 0x27cd78d0, size 4, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd78d0 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8810, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 8, token = 0xfffd7ce302c0, desc = 0xffffdc5ab5b0
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8d50 (mpi-address: 0x27cd8d50) to global_dynamic_window = 667647248 on slave_token 0x27cd7970, size 8, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd7970 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8d50, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xfffd7ce30240, desc = 0xfffd7ce301e8
1/2: _gfortran_caf_register(1240) Attach mem 0x27cd8dd0 (mpi-address: 0x27cd8dd0) to global_dynamic_window = 667647248 on slave_token 0x27cd8010, size 20, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffd7ce301e8 (mpi-address: 281464187060712) to global_dynamic_window 667647248 at address 0x27cd8018, ierr = 0.
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 0
1/2: _gfortran_caf_register(1257) Slave token 0x27cd8010 on exit: mpi_caf_slave_token_t { memptr: 0x27cd8dd0, desc: 0xfffd7ce301e8 }
1/2: _gfortran_caf_register(1191) size = 40, type = 8, token = 0xfffd7ce302a0, desc = 0xfffd7ce30248
1/2: _gfortran_caf_register(1240) Attach mem 0x27cb5e80 (mpi-address: 0x27cb5e80) to global_dynamic_window = 667647248 on slave_token 0x27cd80b0, size 40, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffd7ce30248 (mpi-address: 281464187060808) to global_dynamic_window 667647248 at address 0x27cd80b8, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x27cd80b0 on exit: mpi_caf_slave_token_t { memptr: 0x27cb5e80, desc: 0xfffd7ce30248 }
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 1, elem size in bytes: dst_size = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_COMPONENT
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730957488 (730957488)) <- 0xffffdeca0374, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
1/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 1)
1/2: _gfortran_caf_send_by_ref(5945) remote_image = 0, offset = 0, remote_mem = 0xfffd7ce30108
1/2: _gfortran_caf_send_by_ref(6299) src_rank: 0
1/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 1, elem size in bytes: dst_size = 1
1/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
1/2: send_for_ref(5337) [next == NULL]: src_size = 1, ref_type = CAF_REF_COMPONENT
1/2: put_data(5219) (global_win: 27cb7d10, image: 1, offset: 667780624 (667780624)) <- 0xffffdc5ab88f, num: 1, size 1 -> 1, dst type 1(1), src type 1(1)
2/2: put_data(5231) sr[] = 2, num = 1, num bytes = 4
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 0
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 1, elem size in bytes: dst_size = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_COMPONENT
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958832 (730958832)) <- 0xffffdeca0368, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
[1678368556.681840] [host-10-208-134-176:560336:0]       rma_send.c:277  UCX  ERROR remote memory is unreachable (remote md_map 0x0)
   37.0419998
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 0
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 1, elem size in bytes: dst_size = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_COMPONENT
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958832 (730958832)) <- 0xffffdeca0374, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9dcb0
   37.041999816894531
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 0
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 1, elem size in bytes: dst_size = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_COMPONENT
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730957488 (730957488)) <- 0xffffdeca0368, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9dcb0
   37.041999816894531
   37.0419998
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0x2b918c70
2/2: _gfortran_caf_send_by_ref(6013) remote desc fetch from win 730917488, offset = 224
2/2: _gfortran_caf_send_by_ref(6027) remote desc rank: 1 (ref_rank: 1)
2/2: _gfortran_caf_send_by_ref(6031) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_send_by_ref(6108) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 1
2/2: _gfortran_caf_send_by_ref(6302) src_dim[0] = (1, 5)
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 5, elem size in bytes: dst_size = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 224, ds_glb = 1, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: send_for_ref(5534) remote desc rank: 1 (ref_rank: 1)
2/2: send_for_ref(5538) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: send_for_ref(5544) array_ref_dst[0] = CAF_ARR_REF_FULL := array_ref_src[0] = CAF_ARR_REF_FULL2/2: send_for_ref(5601) CAF_ARR_REF_FULL: src_stride = 1, dst_stride = 1
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958960 (730958960)) <- 0xffffdeca03a0, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
2/2: put_data(5231) sr[] = 51, num = 1, num bytes = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 1] src_index = 1, dst_offset = 4, desc_offset = 4, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958964 (730958964)) <- 0xffffdeca03a4, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
2/2: put_data(5231) sr[] = 102, num = 1, num bytes = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 2] src_index = 2, dst_offset = 8, desc_offset = 8, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958968 (730958968)) <- 0xffffdeca03a8, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
2/2: put_data(5231) sr[] = 51, num = 1, num bytes = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 3] src_index = 3, dst_offset = 12, desc_offset = 12, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958972 (730958972)) <- 0xffffdeca03ac, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
2/2: put_data(5231) sr[] = 154, num = 1, num bytes = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 4] src_index = 4, dst_offset = 16, desc_offset = 16, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958976 (730958976)) <- 0xffffdeca03b0, num: 1, size 4 -> 4, dst type 3(4), src type 3(4)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 4
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0x2b8f75d0
2/2: _gfortran_caf_send_by_ref(6013) remote desc fetch from win 730917488, offset = 320
   5.09999990       4.19999981       3.29999995       2.40000010       1.50000000
2/2: _gfortran_caf_send_by_ref(6027) remote desc rank: 1 (ref_rank: 1)
2/2: _gfortran_caf_send_by_ref(6031) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_send_by_ref(6108) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 1
2/2: _gfortran_caf_send_by_ref(6302) src_dim[0] = (1, 5)
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 5, elem size in bytes: dst_size = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 320, ds_glb = 1, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: send_for_ref(5534) remote desc rank: 1 (ref_rank: 1)
2/2: send_for_ref(5538) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: send_for_ref(5544) array_ref_dst[0] = CAF_ARR_REF_FULL := array_ref_src[0] = CAF_ARR_REF_FULL2/2: send_for_ref(5601) CAF_ARR_REF_FULL: src_stride = 1, dst_stride = 1
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822096 (730822096)) <- 0xffffdeca0378, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
[host-10-208-134-176:560336] *** An error occurred in MPI_Put
[host-10-208-134-176:560336] *** reported by process [1836187649,0]
[host-10-208-134-176:560336] *** on win ucx window 5
[host-10-208-134-176:560336] *** MPI_ERR_OTHER: known error not in list
[host-10-208-134-176:560336] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[host-10-208-134-176:560336] ***    and potentially your MPI job)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 1] src_index = 1, dst_offset = 8, desc_offset = 8, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822104 (730822104)) <- 0xffffdeca0380, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 2] src_index = 2, dst_offset = 16, desc_offset = 16, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822112 (730822112)) <- 0xffffdeca0388, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 3] src_index = 3, dst_offset = 24, desc_offset = 24, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822120 (730822120)) <- 0xffffdeca0390, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 4] src_index = 4, dst_offset = 32, desc_offset = 32, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822128 (730822128)) <- 0xffffdeca0398, num: 1, size 8 -> 8, dst type 3(8), src type 3(8)
2/2: put_data(5231) sr[] = 0, num = 1, num bytes = 8
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0x2b8f75d0
2/2: _gfortran_caf_send_by_ref(6013) remote desc fetch from win 730917488, offset = 320
2/2: _gfortran_caf_send_by_ref(6027) remote desc rank: 1 (ref_rank: 1)
2/2: _gfortran_caf_send_by_ref(6031) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_send_by_ref(6108) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5
   5.0999999046325684        4.1999998092651367        3.2999999523162842        2.4000000953674316        1.5000000000000000
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 1
2/2: _gfortran_caf_send_by_ref(6302) src_dim[0] = (1, 5)
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 5, elem size in bytes: dst_size = 8
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 320, ds_glb = 1, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: send_for_ref(5534) remote desc rank: 1 (ref_rank: 1)
2/2: send_for_ref(5538) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: send_for_ref(5544) array_ref_dst[0] = CAF_ARR_REF_FULL := array_ref_src[0] = CAF_ARR_REF_FULL2/2: send_for_ref(5601) CAF_ARR_REF_FULL: src_stride = 1, dst_stride = 1
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822096 (730822096)) <- 0xffffdeca03a0, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 1] src_index = 1, dst_offset = 8, desc_offset = 8, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822104 (730822104)) <- 0xffffdeca03a4, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 2] src_index = 2, dst_offset = 16, desc_offset = 16, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822112 (730822112)) <- 0xffffdeca03a8, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 3] src_index = 3, dst_offset = 24, desc_offset = 24, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822120 (730822120)) <- 0xffffdeca03ac, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 4] src_index = 4, dst_offset = 32, desc_offset = 32, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 4, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730822128 (730822128)) <- 0xffffdeca03b0, num: 1, size 4 -> 8, dst type 3(8), src type 3(4)
2/2: put_data(5280) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xffffdec9d450
2/2: _gfortran_caf_send_by_ref(5934) Entering send_by_ref(may_require_tmp = 0, dst_type = 3)
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0xffff89cd02c8
2/2: _gfortran_caf_send_by_ref(5945) remote_image = 1, offset = 0, remote_mem = 0x2b918c70
2/2: _gfortran_caf_send_by_ref(6013) remote desc fetch from win 730917488, offset = 224
2/2: _gfortran_caf_send_by_ref(6027) remote desc rank: 1 (ref_rank: 1)
   5.0999999046325684        4.1999998092651367        3.2999999523162842        2.4000000953674316        1.5000000000000000
2/2: _gfortran_caf_send_by_ref(6031) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_send_by_ref(6108) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5
2/2: _gfortran_caf_send_by_ref(6299) src_rank: 1
2/2: _gfortran_caf_send_by_ref(6302) src_dim[0] = (1, 5)
2/2: _gfortran_caf_send_by_ref(6334) calling send_for_ref. num elems: size = 5, elem size in bytes: dst_size = 4
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 0, desc_glb = 0
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 224, ds_glb = 1, desc_glb = 0
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: send_for_ref(5534) remote desc rank: 1 (ref_rank: 1)
2/2: send_for_ref(5538) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: send_for_ref(5544) array_ref_dst[0] = CAF_ARR_REF_FULL := array_ref_src[0] = CAF_ARR_REF_FULL2/2: send_for_ref(5601) CAF_ARR_REF_FULL: src_stride = 1, dst_stride = 1
2/2: send_for_ref(5329) Entering send_for_ref: [i = 0] src_index = 0, dst_offset = 0, desc_offset = 0, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958960 (730958960)) <- 0xffffdeca0378, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 1] src_index = 1, dst_offset = 4, desc_offset = 4, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958964 (730958964)) <- 0xffffdeca0380, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 2] src_index = 2, dst_offset = 8, desc_offset = 8, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958968 (730958968)) <- 0xffffdeca0388, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 3] src_index = 3, dst_offset = 12, desc_offset = 12, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958972 (730958972)) <- 0xffffdeca0390, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9d450
2/2: send_for_ref(5329) Entering send_for_ref: [i = 4] src_index = 4, dst_offset = 16, desc_offset = 16, ds_glb = 1, desc_glb = 1
2/2: send_for_ref(5337) [next == NULL]: src_size = 8, ref_type = CAF_REF_ARRAY
2/2: put_data(5219) (global_win: 2b8f7dd0, image: 2, offset: 730958976 (730958976)) <- 0xffffdeca0398, num: 1, size 8 -> 4, dst type 3(4), src type 3(8)
2/2: put_data(5280) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xffffdec9d450
2/2: _gfortran_caf_sync_all(1591) Entering sync all.
   5.09999990       4.19999981       3.29999995       2.40000010       1.50000000
Error: Command:
   `/root/hpcrunner/software/mpi/hmpi1.1.1-kgcc9.3.1/1.1.1/bin/mpiexec -n 2 /root/hpcrunner/tmp/OpenCoarrays-2.10.1/opencoarrays-build/bin/OpenCoarrays-2.10.1-tests/alloc_comp_send_convert_nums`
failed to run.

20/81 Test #14: alloc_comp_get_convert_nums ......................***Failed  Required regular expression not found. Regex=[Test passed.
]  0.10 sec
2/2: _gfortran_caf_register(1191) size = 448, type = 0, token = 0x43f318, desc = 0xfffffb3b5c10
1/2: _gfortran_caf_init(950) The mpi memory model is: unified  (0x0, 1).
1/2: _gfortran_caf_register(1191) size = 448, type = 0, token = 0x43f318, desc = 0xffffca9e4550
2/2: _gfortran_caf_register(1307) Token 0xd54eaa0 on exit: mpi_caf_token_t { (local_)memptr: 0xffff701f02c8 (size: 448), memptr_win: 223668928 }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xfffffb3b5bf0, desc = 0xfffffb3b5990
1/2: _gfortran_caf_register(1307) Token 0x3f71e980 on exit: mpi_caf_token_t { (local_)memptr: 0xfffc40130108 (size: 448), memptr_win: 1064429984 }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffca9e4530, desc = 0xffffca9e42d0
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f7277b0 (size: 16, mpi-address: 0x3f7277b0) to global_dynamic_window = 1064336688
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557740 (size: 16, mpi-address: 0xd557740) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557740 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xfffffb3b5bf8, desc = 0xfffffb3b59c0
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557780 (size: 16, mpi-address: 0xd557780) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557780 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1257) Slave token 0x3f7277b0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffca9e4538, desc = 0xffffca9e4300
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f727850 (size: 16, mpi-address: 0x3f727850) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f727850 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffca9e4540, desc = 0xffffca9e4330
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xfffffb3b5c00, desc = 0xfffffb3b59f0
2/2: _gfortran_caf_register(1216) Attach slave token 0xd5577c0 (size: 16, mpi-address: 0xd5577c0) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd5577c0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xfffffb3b5c08, desc = 0xfffffb3b5a20
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557860 (size: 16, mpi-address: 0xd557860) to global_dynamic_window = 223575584
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f7278f0 (size: 16, mpi-address: 0x3f7278f0) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f7278f0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 7, token = 0xffffca9e4548, desc = 0xffffca9e4360
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f727990 (size: 16, mpi-address: 0x3f727990) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f727990 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffca9e4408, desc = 0xffffca9e43b0
2/2: _gfortran_caf_register(1257) Slave token 0xd557860 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xfffffb3b5ac8, desc = 0xfffffb3b5a70
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557dc0 (size: 16, mpi-address: 0xd557dc0) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557dc0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xfffffb3b5b28, desc = 0xfffffb3b5ad0
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f727ef0 (size: 16, mpi-address: 0x3f727ef0) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f727ef0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffca9e4468, desc = 0xffffca9e4410
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f727f90 (size: 16, mpi-address: 0x3f727f90) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f727f90 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557e60 (size: 16, mpi-address: 0xd557e60) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557e60 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xfffffb3b5b88, desc = 0xfffffb3b5b30
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557f00 (size: 16, mpi-address: 0xd557f00) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557f00 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xfffffb3b5be8, desc = 0xfffffb3b5b90
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffca9e44c8, desc = 0xffffca9e4470
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f728030 (size: 16, mpi-address: 0x3f728030) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f728030 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 0, type = 7, token = 0xffffca9e4528, desc = 0xffffca9e44d0
1/2: _gfortran_caf_register(1216) Attach slave token 0x3f7280d0 (size: 16, mpi-address: 0x3f7280d0) to global_dynamic_window = 1064336688
1/2: _gfortran_caf_register(1257) Slave token 0x3f7280d0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1216) Attach slave token 0xd557fa0 (size: 16, mpi-address: 0xd557fa0) to global_dynamic_window = 223575584
2/2: _gfortran_caf_register(1257) Slave token 0xd557fa0 on exit: mpi_caf_slave_token_t { memptr: (nil), desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 1, type = 8, token = 0xffff701f0468, desc = 0xfffffb3b5948
1/2: _gfortran_caf_register(1191) size = 1, type = 8, token = 0xfffc401302a8, desc = 0xffffca9e4288
1/2: _gfortran_caf_register(1240) Attach mem 0x3f728630 (mpi-address: 0x3f728630) to global_dynamic_window = 1064336688 on slave_token 0x3f7277b0, size 1, ierr: 0
2/2: _gfortran_caf_register(1240) Attach mem 0xd558500 (mpi-address: 0xd558500) to global_dynamic_window = 223575584 on slave_token 0xd557740, size 1, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0xd557740 on exit: mpi_caf_slave_token_t { memptr: 0xd558500, desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xffff701f0470, desc = 0xfffffb3b5948
2/2: _gfortran_caf_register(1240) Attach mem 0xd558580 (mpi-address: 0xd558580) to global_dynamic_window = 223575584 on slave_token 0xd557780, size 4, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0xd557780 on exit: mpi_caf_slave_token_t { memptr: 0xd558580, desc: (nil) }
1/2: _gfortran_caf_register(1257) Slave token 0x3f7277b0 on exit: mpi_caf_slave_token_t { memptr: 0x3f728630, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xfffc401302b0, desc = 0xffffca9e4288
1/2: _gfortran_caf_register(1240) Attach mem 0x3f7286b0 (mpi-address: 0x3f7286b0) to global_dynamic_window = 1064336688 on slave_token 0x3f727850, size 4, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x3f727850 on exit: mpi_caf_slave_token_t { memptr: 0x3f7286b0, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 5, type = 8, token = 0xfffc40130180, desc = 0xfffc40130128
2/2: _gfortran_caf_register(1191) size = 5, type = 8, token = 0xffff701f0340, desc = 0xffff701f02e8
2/2: _gfortran_caf_register(1240) Attach mem 0xd558600 (mpi-address: 0xd558600) to global_dynamic_window = 223575584 on slave_token 0xd557dc0, size 5, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff701f02e8 (mpi-address: 281472562823912) to global_dynamic_window 223575584 at address 0xd557dc8, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0xd557dc0 on exit: mpi_caf_slave_token_t { memptr: 0xd558600, desc: 0xffff701f02e8 }
2/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xffff701f03a0, desc = 0xffff701f0348
1/2: _gfortran_caf_register(1240) Attach mem 0x3f728730 (mpi-address: 0x3f728730) to global_dynamic_window = 1064336688 on slave_token 0x3f727ef0, size 5, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffc40130128 (mpi-address: 281458871828776) to global_dynamic_window 1064336688 at address 0x3f727ef8, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x3f727ef0 on exit: mpi_caf_slave_token_t { memptr: 0x3f728730, desc: 0xfffc40130128 }
1/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xfffc401301e0, desc = 0xfffc40130188
1/2: _gfortran_caf_register(1240) Attach mem 0x3f7287b0 (mpi-address: 0x3f7287b0) to global_dynamic_window = 1064336688 on slave_token 0x3f727f90, size 20, ierr: 0
2/2: _gfortran_caf_register(1240) Attach mem 0xd558680 (mpi-address: 0xd558680) to global_dynamic_window = 223575584 on slave_token 0xd557e60, size 20, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff701f0348 (mpi-address: 281472562824008) to global_dynamic_window 223575584 at address 0xd557e68, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0xd557e60 on exit: mpi_caf_slave_token_t { memptr: 0xd558680, desc: 0xffff701f0348 }
2/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xffff701f0478, desc = 0xfffffb3b5948
2/2: _gfortran_caf_register(1240) Attach mem 0xd558700 (mpi-address: 0xd558700) to global_dynamic_window = 223575584 on slave_token 0xd5577c0, size 4, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0xd5577c0 on exit: mpi_caf_slave_token_t { memptr: 0xd558700, desc: (nil) }
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffc40130188 (mpi-address: 281458871828872) to global_dynamic_window 1064336688 at address 0x3f727f98, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x3f727f90 on exit: mpi_caf_slave_token_t { memptr: 0x3f7287b0, desc: 0xfffc40130188 }
1/2: _gfortran_caf_register(1191) size = 4, type = 8, token = 0xfffc401302b8, desc = 0xffffca9e4288
1/2: _gfortran_caf_register(1240) Attach mem 0x3f728830 (mpi-address: 0x3f728830) to global_dynamic_window = 1064336688 on slave_token 0x3f7278f0, size 4, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x3f7278f0 on exit: mpi_caf_slave_token_t { memptr: 0x3f728830, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 8, type = 8, token = 0xfffc401302c0, desc = 0xffffca9e4288
2/2: _gfortran_caf_register(1191) size = 8, type = 8, token = 0xffff701f0480, desc = 0xfffffb3b5948
2/2: _gfortran_caf_register(1240) Attach mem 0xd558c40 (mpi-address: 0xd558c40) to global_dynamic_window = 223575584 on slave_token 0xd557860, size 8, ierr: 0
2/2: _gfortran_caf_register(1257) Slave token 0xd557860 on exit: mpi_caf_slave_token_t { memptr: 0xd558c40, desc: (nil) }
2/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xffff701f0400, desc = 0xffff701f03a8
2/2: _gfortran_caf_register(1240) Attach mem 0xd558cc0 (mpi-address: 0xd558cc0) to global_dynamic_window = 223575584 on slave_token 0xd557f00, size 20, ierr: 0
1/2: _gfortran_caf_register(1240) Attach mem 0x3f728d70 (mpi-address: 0x3f728d70) to global_dynamic_window = 1064336688 on slave_token 0x3f727990, size 8, ierr: 0
1/2: _gfortran_caf_register(1257) Slave token 0x3f727990 on exit: mpi_caf_slave_token_t { memptr: 0x3f728d70, desc: (nil) }
1/2: _gfortran_caf_register(1191) size = 20, type = 8, token = 0xfffc40130240, desc = 0xfffc401301e8
1/2: _gfortran_caf_register(1240) Attach mem 0x3f728df0 (mpi-address: 0x3f728df0) to global_dynamic_window = 1064336688 on slave_token 0x3f728030, size 20, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffc401301e8 (mpi-address: 281458871828968) to global_dynamic_window 1064336688 at address 0x3f728038, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x3f728030 on exit: mpi_caf_slave_token_t { memptr: 0x3f728df0, desc: 0xfffc401301e8 }
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff701f03a8 (mpi-address: 281472562824104) to global_dynamic_window 223575584 at address 0xd557f08, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0xd557f00 on exit: mpi_caf_slave_token_t { memptr: 0xd558cc0, desc: 0xffff701f03a8 }
2/2: _gfortran_caf_register(1191) size = 40, type = 8, token = 0xffff701f0460, desc = 0xffff701f0408
2/2: _gfortran_caf_register(1240) Attach mem 0xd537620 (mpi-address: 0xd537620) to global_dynamic_window = 223575584 on slave_token 0xd557fa0, size 40, ierr: 0
2/2: _gfortran_caf_register(1250) Attached descriptor 0xffff701f0408 (mpi-address: 281472562824200) to global_dynamic_window 223575584 at address 0xd557fa8, ierr = 0.
2/2: _gfortran_caf_register(1257) Slave token 0xd557fa0 on exit: mpi_caf_slave_token_t { memptr: 0xd537620, desc: 0xffff701f0408 }
1/2: _gfortran_caf_register(1191) size = 40, type = 8, token = 0xfffc401302a0, desc = 0xfffc40130248
1/2: _gfortran_caf_register(1240) Attach mem 0x3f705ea0 (mpi-address: 0x3f705ea0) to global_dynamic_window = 1064336688 on slave_token 0x3f7280d0, size 40, ierr: 0
1/2: _gfortran_caf_register(1250) Attached descriptor 0xfffc40130248 (mpi-address: 281458871829064) to global_dynamic_window 1064336688 at address 0x3f7280d8, ierr = 0.
1/2: _gfortran_caf_register(1257) Slave token 0x3f7280d0 on exit: mpi_caf_slave_token_t { memptr: 0x3f705ea0, desc: 0xfffc40130248 }
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4628, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
1/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 0, global_rank = 0.
1/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xffffca9e2f68, type = 0,  offset = 0, remote_mem = 0xfffc40130108, global_win(data, desc)) = (0, 0)
1/2: _gfortran_caf_get_by_ref(4711) get(custom_token 1064429984): remote_memptr(old) = 0xfffc40130108, remote_memptr(new) = 0x3f728630, offset = 0
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558700, offset = 16
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 0
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4628 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4112) caf_offset = 432
2/2: get_data(3962) 0xfffffb3b5be4 = global_win(2) offset: 223708928 (0xd558700) of size 4 -> 4, dst type 3(4), src type 3(4)
1/2: _gfortran_caf_get_by_ref(5186) dst_rank: 0
1/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
1/2: get_for_ref(4100) caf_ref = 0xffffca9e2f68 (type = 0), sr_offset = 0, sr = 0xfffc40130108, rdesc = (nil), desc_offset = 0, src = 0xffffca9e4550, sr_glb = 0, desc_glb = 0, src_dim = 0
1/2: get_for_ref(4112) caf_offset = 416
1/2: get_data(3962) 0xffffca9e456f = global_win(1) offset: 1064470064 (0x3f728630) of size 1 -> 1, dst type 1(1), src type 1(1)
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b47c0, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558c40, offset = 24
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 0
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b47c0 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4112) caf_offset = 440
2/2: get_data(3962) 0xfffffb3b5bd8 = global_win(2) offset: 223710272 (0xd558c40) of size 8 -> 8, dst type 3(8), src type 3(8)
[1678368560.166262] [host-10-208-134-176:560672:0]       rma_send.c:377  UCX  ERROR remote memory is unreachable (remote md_map 0x0)
   37.0419998
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4958, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558700, offset = 16
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 0
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4958 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4112) caf_offset = 432
2/2: get_data(3962) 0xfffffb3b5bd8 = global_win(2) offset: 223708928 (0xd558700) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b35f0
   37.041999816894531
2/2: get_data(4019) srh[0] = 2, ierr = 0
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4af0, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558c40, offset = 24
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 0
   37.041999816894531
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4af0 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4112) caf_offset = 440
2/2: get_data(3962) 0xfffffb3b5be4 = global_win(2) offset: 223710272 (0xd558c40) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b35f0
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4c88, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558cc0, offset = 224
   37.0419998
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4e20, type = 1,  offset = 0, remote_mem = 0xd558cc0, global_win(data, desc)) = (1, 0)
2/2: _gfortran_caf_get_by_ref(4757) remote desc fetch from win 223668928, offset = 224, ref_rank = 1
2/2: _gfortran_caf_get_by_ref(4784) remote desc rank: 1, base_addr: 0xd558cc0
2/2: _gfortran_caf_get_by_ref(4787) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_get_by_ref(4865) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5, in_array_ref = 0, arr_ext_fixed = 0, realloc_required = 0
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 1
2/2: _gfortran_caf_get_by_ref(5189) dst_dim[0] = (1, 5)
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4c88 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 0, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 224, src = (nil), sr_glb = 1, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4282) Fetching remote data.
2/2: get_for_ref(4300) remote desc rank: 1, base: 0xd558cc0
2/2: get_for_ref(4304) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 0, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 0, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c10 = global_win(2) offset: 223710400 (0xd558cc0) of size 4 -> 4, dst type 3(4), src type 3(4)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 4, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 4, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c14 = global_win(2) offset: 223710404 (0xd558cc4) of size 4 -> 4, dst type 3(4), src type 3(4)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 8, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 8, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c18 = global_win(2) offset: 223710408 (0xd558cc8) of size 4 -> 4, dst type 3(4), src type 3(4)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 12, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 12, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c1c = global_win(2) offset: 223710412 (0xd558ccc) of size 4 -> 4, dst type 3(4), src type 3(4)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4e20 (type = 1), sr_offset = 16, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 16, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c20 = global_win(2) offset: 223710416 (0xd558cd0) of size 4 -> 4, dst type 3(4), src type 3(4)
[host-10-208-134-176:560672] *** An error occurred in MPI_Get
[host-10-208-134-176:560672] *** reported by process [1872887809,0]
[host-10-208-134-176:560672] *** on win ucx window 5
[host-10-208-134-176:560672] *** MPI_ERR_OTHER: known error not in list
[host-10-208-134-176:560672] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[host-10-208-134-176:560672] ***    and potentially your MPI job)
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b4fb8, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd537620, offset = 320
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b5150, type = 1,  offset = 0, remote_mem = 0xd537620, global_win(data, desc)) = (1, 0)
2/2: _gfortran_caf_get_by_ref(4757) remote desc fetch from win 223668928, offset = 320, ref_rank = 1
2/2: _gfortran_caf_get_by_ref(4784) remote desc rank: 1, base_addr: 0xd537620
2/2: _gfortran_caf_get_by_ref(4787) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_get_by_ref(4865) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5, in_array_ref = 0, arr_ext_fixed = 0, realloc_required = 0
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 1
2/2: _gfortran_caf_get_by_ref(5189) dst_dim[0] = (1, 5)
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b4fb8 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 0, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 320, src = (nil), sr_glb = 1, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4282) Fetching remote data.
2/2: get_for_ref(4300) remote desc rank: 1, base: 0xd537620
2/2: get_for_ref(4304) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 0, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 0, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5be8 = global_win(2) offset: 223573536 (0xd537620) of size 8 -> 8, dst type 3(8), src type 3(8)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 8, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 8, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5bf0 = global_win(2) offset: 223573544 (0xd537628) of size 8 -> 8, dst type 3(8), src type 3(8)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 16, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 16, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5bf8 = global_win(2) offset: 223573552 (0xd537630) of size 8 -> 8, dst type 3(8), src type 3(8)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 24, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 24, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c00 = global_win(2) offset: 223573560 (0xd537638) of size 8 -> 8, dst type 3(8), src type 3(8)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5150 (type = 1), sr_offset = 32, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 32, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c08 = global_win(2) offset: 223573568 (0xd537640) of size 8 -> 8, dst type 3(8), src type 3(8)
   5.09999990       4.19999981       3.29999995       2.40000010       1.50000000
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b52e8, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd558cc0, offset = 224
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b5480, type = 1,  offset = 0, remote_mem = 0xd558cc0, global_win(data, desc)) = (1, 0)
2/2: _gfortran_caf_get_by_ref(4757) remote desc fetch from win 223668928, offset = 224, ref_rank = 1
2/2: _gfortran_caf_get_by_ref(4784) remote desc rank: 1, base_addr: 0xd558cc0
2/2: _gfortran_caf_get_by_ref(4787) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_get_by_ref(4865) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5, in_array_ref = 0, arr_ext_fixed = 0, realloc_required = 0
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 1
2/2: _gfortran_caf_get_by_ref(5189) dst_dim[0] = (1, 5)
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b52e8 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 0, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 224, src = (nil), sr_glb = 1, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4282) Fetching remote data.
2/2: get_for_ref(4300) remote desc rank: 1, base: 0xd558cc0
2/2: get_for_ref(4304) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 0, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 0, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
   5.0999999046325684        4.1999998092651367        3.2999999523162842        2.4000000953674316        1.5000000000000000
2/2: get_data(3962) 0xfffffb3b5be8 = global_win(2) offset: 223710400 (0xd558cc0) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 51, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 4, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 4, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5bf0 = global_win(2) offset: 223710404 (0xd558cc4) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 102, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 8, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 8, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5bf8 = global_win(2) offset: 223710408 (0xd558cc8) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 51, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 12, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 12, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c00 = global_win(2) offset: 223710412 (0xd558ccc) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 154, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5480 (type = 1), sr_offset = 16, sr = 0xd558cc0, rdesc = 0xffff701f02c8, desc_offset = 16, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c08 = global_win(2) offset: 223710416 (0xd558cd0) of size 4 -> 8, dst type 3(8), src type 3(4)
2/2: get_data(4012) type/kind convert 1 items: type 3(4) -> type 3(8), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: _gfortran_caf_get_by_ref(4672) Entering get_by_ref(may_require_tmp = 0), win_rank = 1, global_rank = 1.
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b5618, type = 0,  offset = 0, remote_mem = 0xffff701f02c8, global_win(data, desc)) = (0, 0)
2/2: _gfortran_caf_get_by_ref(4711) get(custom_token 223668928): remote_memptr(old) = 0xffff701f02c8, remote_memptr(new) = 0xd537620, offset = 320
   5.0999999046325684        4.1999998092651367        3.2999999523162842        2.4000000953674316        1.5000000000000000
2/2: _gfortran_caf_get_by_ref(4680) caf_ref = 0xfffffb3b57b0, type = 1,  offset = 0, remote_mem = 0xd537620, global_win(data, desc)) = (1, 0)
2/2: _gfortran_caf_get_by_ref(4757) remote desc fetch from win 223668928, offset = 320, ref_rank = 1
2/2: _gfortran_caf_get_by_ref(4784) remote desc rank: 1, base_addr: 0xd537620
2/2: _gfortran_caf_get_by_ref(4787) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: _gfortran_caf_get_by_ref(4865) i = 0, array_ref = CAF_ARR_REF_FULL, delta = 5, in_array_ref = 0, arr_ext_fixed = 0, realloc_required = 0
2/2: _gfortran_caf_get_by_ref(5186) dst_rank: 1
2/2: _gfortran_caf_get_by_ref(5189) dst_dim[0] = (1, 5)
2/2: _gfortran_caf_get_by_ref(5194) get_by_ref() calling get_for_ref.
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b5618 (type = 0), sr_offset = 0, sr = 0xffff701f02c8, rdesc = (nil), desc_offset = 0, src = 0xfffffb3b5c10, sr_glb = 0, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 0, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 320, src = (nil), sr_glb = 1, desc_glb = 0, src_dim = 0
2/2: get_for_ref(4282) Fetching remote data.
2/2: get_for_ref(4300) remote desc rank: 1, base: 0xd537620
2/2: get_for_ref(4304) remote desc dim[0] = (lb=1, ub=5, stride=1)
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 0, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 0, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c10 = global_win(2) offset: 223573536 (0xd537620) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 8, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 8, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c14 = global_win(2) offset: 223573544 (0xd537628) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 16, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 16, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c18 = global_win(2) offset: 223573552 (0xd537630) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 24, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 24, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c1c = global_win(2) offset: 223573560 (0xd537638) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: get_for_ref(4100) caf_ref = 0xfffffb3b57b0 (type = 1), sr_offset = 32, sr = 0xd537620, rdesc = 0xffff701f02c8, desc_offset = 32, src = 0xfffffb3b3568, sr_glb = 1, desc_glb = 1, src_dim = 1
2/2: get_data(3962) 0xfffffb3b5c20 = global_win(2) offset: 223573568 (0xd537640) of size 8 -> 4, dst type 3(4), src type 3(8)
2/2: get_data(4012) type/kind convert 1 items: type 3(8) -> type 3(4), local buffer: 0xfffffb3b2d90
2/2: get_data(4019) srh[0] = 0, ierr = 0
2/2: _gfortran_caf_sync_all(1591) Entering sync all.
   5.09999990       4.19999981       3.29999995       2.40000010       1.50000000
Error: Command:
   `/root/hpcrunner/software/mpi/hmpi1.1.1-kgcc9.3.1/1.1.1/bin/mpiexec -n 2 /root/hpcrunner/tmp/OpenCoarrays-2.10.1/opencoarrays-build/bin/OpenCoarrays-2.10.1-tests/alloc_comp_get_convert_nums`
failed to run.