Unclear if two-pass compilation on NV GPU supports old versions of NVHPC

[sbryngelson]

Unclear if #581 two-pass compilation of nvidia GPU builds works on nvhpc 22.X or 23.X (known working on 24.X). Should first check if this is the case or not (can change this line to FALSE in CMakeLists.txt:
set(NVHPC_USE_TWO_PASS_IPO TRUE)

and if this feature does not work, add a check in CMakeLists.txt to disable this feature if an old nvhpc is found.

Update: NVHPC 23.11 works. Older versions do not.

[sbryngelson]

Mode of failure using nvhpc22.9 on Bridges2 occurs at runtime (well, test-time):

[I]v025: bryngel-startup-proj/MFC $ ./mfc.sh test -j 1 --gpu --no-build -- -c bridges2
mfc: OK > (venv) Entered the Python 3.8.6 virtual environment (>= 3.8).

      .=++*:          -+*+=.        | bryngel@v025.ib.bridges2.psc.edu [Linux]
     :+   -*-        ==   =* .      | ----------------------------------------
   :*+      ==      ++    .+-       |
  :*##-.....:*+   .#%+++=--+=:::.   | --jobs 1
  -=-++-======#=--**+++==+*++=::-:. | --mpi --gpu --no-debug --no-gcov --no-unified
 .:++=----------====+*= ==..:%..... |
  .:-=++++===--==+=-+=   +.  :=     |
  +#=::::::::=%=. -+:    =+   *:    | ----------------------------------------------------------
 .*=-=*=..    :=+*+:      -...--    | $ ./mfc.sh (build, run, test, clean, count, packer) --help

 Build | syscheck, pre_process, and simulation | Generic Build



 Test pre_process and simulation | from D79C3E6F to AB6E5D10 (277 tests)

    tests/UUID     (s)      Summary

   mfc: OK > (venv) Entered the Python 3.8.6 virtual environment (>= 3.8).

      .=++*:          -+*+=.        | bryngel@v025.ib.bridges2.psc.edu [Linux]
     :+   -*-        ==   =* .      | ----------------------------------------
   :*+      ==      ++    .+-       |
  :*##-.....:*+   .#%+++=--+=:::.   | --jobs 1
  -=-++-======#=--**+++==+*++=::-:. | --mpi --gpu --no-debug --no-gcov --no-unified
 .:++=----------====+*= ==..:%..... | --targets pre_process and simulation
  .:-=++++===--==+=-+=   +.  :=     |
  +#=::::::::=%=. -+:    =+   *:    | ----------------------------------------------------------
 .*=-=*=..    :=+*+:      -...--    | $ ./mfc.sh (build, run, test, clean, count, packer) --help

 Acquiring /ocean/projects/phy210041p/bryngel/MFC/tests/D79C3E6F/case.py...
 Acquiring /ocean/projects/phy210041p/bryngel/MFC/tests/D79C3E6F/case.py...
 Build | syscheck, syscheck, pre_process, and simulation | Generic Build


 Run
   Using queue system Interactive.
   Using baked-in template for bridges2.
   Generating input files for syscheck...

     Generating syscheck.inp:
       INFO: Forwarded 0/90 parameters.

   Generating input files for pre_process...

     Generating pre_process.inp:
       INFO: Forwarded 64/90 parameters.

   Generating input files for simulation...

     Generating simulation.inp:
       INFO: Forwarded 57/90 parameters.

   $ /bin/bash /ocean/projects/phy210041p/bryngel/MFC/tests/D79C3E6F/MFC.sh

+-----------------------------------------------------------------------------------------------------------+
| MFC case # MFC @ /ocean/projects/phy210041p/bryngel/MFC/tests/D79C3E6F/case.py:                           |
+-----------------------------------------------------------------------------------------------------------+
| * Start-time     12:46:28                            * Start-date     12:46:28                            |
| * Partition      N/A                                 * Walltime       01:00:00                            |
| * Account        N/A                                 * Nodes          1                                   |
| * Job Name       MFC                                 * Engine         interactive                         |
| * QoS            N/A                                 * Binary         N/A                                 |
| * Queue System   Interactive                         * Email          N/A                                 |
+-----------------------------------------------------------------------------------------------------------+

mfc: OK > :) Loading modules:


mfc: OK > :) Running syscheck:

+ mpirun -np 1 /ocean/projects/phy210041p/bryngel/MFC/build/install/ea58b6820d/bin/syscheck
 [TEST] MPI: call mpi_init(ierr)
 [TEST] MPI: call mpi_comm_rank(MPI_COMM_WORLD, rank, ierr)
 [TEST] MPI: call mpi_barrier(MPI_COMM_WORLD, ierr)
 [TEST] MPI: call assert(rank >= 0)
 [TEST] MPI: call mpi_comm_size(MPI_COMM_WORLD, nRanks, ierr)
 [TEST] MPI: call assert(nRanks > 0 .and. rank < nRanks)
 [TEST] ACC: devtype = acc_get_device_type()
 [TEST] ACC: num_devices = acc_get_num_devices(devtype)
 [TEST] ACC: call assert(num_devices > 0)
 [TEST] ACC: call acc_set_device_num(mod(rank, nRanks), devtype)
 [TEST] ACC: allocate(arr(1:N))
 [TEST] ACC: !$acc enter data create(arr(1:N))
 [TEST] ACC: !$acc parallel loop
 [TEST] ACC: !$acc update host(arr(1:N))
 [TEST] ACC: !$acc exit data delete(arr)
 [TEST] MPI: call mpi_barrier(MPI_COMM_WORLD, ierr)
 [TEST] MPI: call mpi_finalize(ierr)

 Syscheck: PASSED.

mfc: OK > :) Running pre_process:

+ mpirun -np 1 /ocean/projects/phy210041p/bryngel/MFC/build/install/25eb1bf60b/bin/pre_process
 Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7)
==== backtrace (tid:  29406) ====
 0 0x0000000000012cf0 __funlockfile()  :0
 1 0x0000000000421c53 m_helper_basic_f_approx_equal_()  /jet/home/bryngel/bryngel-startup-proj/MFC/src/common/m_helper_basic.f90:0
 2 0x000000000047115e m_checker_common_s_check_inputs_common_()  /jet/home/bryngel/bryngel-startup-proj/MFC/src/common/m_checker_common.fpp:56
 3 0x0000000000470a2d m_start_up_s_initialize_mpi_domain_()  /jet/home/bryngel/bryngel-startup-proj/MFC/src/pre_process/m_start_up.fpp:202
 4 0x000000000041a586 MAIN_()  /jet/home/bryngel/bryngel-startup-proj/MFC/src/pre_process/p_main.f90:26
 5 0x0000000000404033 main()  ???:0
 6 0x000000000003ad85 __libc_start_main()  ???:0
 7 0x0000000000403f2e _start()  ???:0
=================================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 29406 on node v025 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

mfc: ERROR > :( /ocean/projects/phy210041p/bryngel/MFC/build/install/25eb1bf60b/bin/pre_process failed with exit code 139.



Error: Submitting batch file for Interactive failed. It can be found here: /ocean/projects/phy210041p/bryngel/MFC/tests/D79C3E6F/MFC.sh. Please check the file for errors.

./mfc.sh: line 55: 29350 Terminated              python3 "$(pwd)/toolchain/main.py" "$@"

mfc: ERROR > main.py finished with a 143 exit code.
mfc: (venv) Exiting the Python virtual environment.

[sbryngelson]

Actually, this also fails (Rogues Gallery, Violet1, A100): nvhpc-hpcx-cuda11/23.7 with two-pass optimization via the same failure mode (passes with two-pass IPO set to false).

[sbryngelson]

nvhpc23.11 works with two-pass IPO

[sbryngelson]

This issue can be resolved by updating modules on common HPC systems like Bridges2 and Delta and disabling IPO/LTO if the detected NVHPC version is less than 23.11