Persistent hanging jobs with Stochastic Tools + Griffin, on Sawtooth

[harterj]

Hi all,

I've recently been using stochastic tools in conjunction with Griffin for providing sensitivity and uncertainty quantification, as well as attempting to develop a surrogate, for a nuclear thermal propulsion project. I've had some success, but I've consistently encountered issues on Sawtooth with simulation initializations, during solves, and at the solve completion. I get hanging jobs frequently, and these are relatively intensive (60 node) jobs. If there is any advice out there, I'd appreciate it. Here are the modules I've been using on sawtooth, loading use.moose and moose-dev:

  1) use.moose                     4) curl/7.61.1-gcc-12.3.0-5t73   7) cmake/3.27.7-gcc-12.3.0-5cfk  10) mpfr/4.2.0-gcc-10.5.0-cec4  13) gcc/12.3.0-gcc-10.5.0-vx2f   16) moose-dev/2023-10-23
  2) flex/2.6.4-gcc-12.3.0-cnsv    5) ncurses/6.1-gcc-12.3.0-7zps   8) patchelf/0.18.0               11) mpc/1.3.1-gcc-10.5.0-mpcb   14) mvapich2/2.3.7-1-gcc-12.3.0
  3) bison/3.0.4-gcc-12.3.0-547w   6) zlib/1.2.13-gcc-12.3.0-c2ww   9) gmp/6.1.2-gcc-10.5.0-wep3     12) zstd/1.4.4-gcc-10.5.0-yue3  15) moose-tools/2023-10-19

Here are a couple examples of the hanging jobs. Thanks in advance if anyone is able to provide some guidance.

Jackson

[r8i2n1.ib0.sawtooth.hpc.inl.gov](http://r8i2n1.ib0.sawtooth.hpc.inl.gov/)
[r8i2n1.ib0.sawtooth.hpc.inl.gov](http://r8i2n1.ib0.sawtooth.hpc.inl.gov/)
[r8i1n6:mpi_rank_1851][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xf813780). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic pattern
s. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r8i1n6:mpi_rank_1833][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x103575b0). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patter
ns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r8i1n6:mpi_rank_1849][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x1049e780). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patter

runner00_griffin0: Beginning fixed point iteration 0
runner00_griffin0: 
runner00_griffin0: Fixed point residual norm after TIMESTEP_BEGIN MultiApps: 5.840389e-02
runner00_griffin0: 
runner00_griffin0: Main app solve:
runner00_griffin0:  0 Nonlinear |R| = 5.840389e-02
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 100
[r5i1n3:mpi_rank_100][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, -1) - process 100: Cannot allocate memory (12)
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 109
[r5i1n3:mpi_rank_109][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, -1) - process 109: Cannot allocate memory (12)
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1156
[r5i4n33:mpi_rank_1156][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1156: Cannot allocate memory (12)
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1192
[r5i4n33:mpi_rank_1192][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1192: Cannot allocate memory (12)
[r5i2n30:mpi_rank_654][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x92794d70). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i2n30:mpi_rank_655][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x92bf6f40). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i2n30:mpi_rank_637][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xae076260). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i2n9:mpi_rank_336][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x91a032f0). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i2n9:mpi_rank_366][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xddfc7a80). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i1n0:mpi_rank_70][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x1107f9510). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i1n0:mpi_rank_77][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x7f700830). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i0n33:mpi_rank_45][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xb9c1c040). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i5n4:mpi_rank_1312][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xb31a39e0). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i5n4:mpi_rank_1341][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x10ba1c390). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i5n0:mpi_rank_1212][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x10c0fc780). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r5i5n0:mpi_rank_1213][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xb29cbf00). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[mpiexec@r5i0n33] HYDU_sock_write (../../../../src/pm/hydra/utils/sock/sock.c:303): write error (Bad file descriptor)
[mpiexec@r5i0n33] HYD_pmcd_pmiserv_send_signal (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy
[mpiexec@r5i0n33] ui_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream
[mpiexec@r5i0n33] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@r5i0n33] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event
[mpiexec@r5i0n33] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:340): process manager error waiting for completion

SamplerFullSolveMultiApp [runner] 8/75 samples complete!
SamplerFullSolveMultiApp [runner] 16/75 samples complete!
SamplerFullSolveMultiApp [runner] 24/75 samples complete!
SamplerFullSolveMultiApp [runner] 32/75 samples complete!
SamplerFullSolveMultiApp [runner] 40/75 samples complete!
SamplerFullSolveMultiApp [runner] 47/75 samples complete!
SamplerFullSolveMultiApp [runner] 54/75 samples complete!
SamplerFullSolveMultiApp [runner] 61/75 samples complete!
SamplerFullSolveMultiApp [runner] 68/75 samples complete!
SamplerFullSolveMultiApp [runner] 75/75 samples complete!
        Finished Executing MultiApps                                                     [19511.66 s] [ 6638 MB]
      Finished Solving Step Batch For SamplerFullSolveMultiApp                           [132004.60 s] [ 6638 MB]
    Finished Executing MultiApps                                                         [132004.60 s] [ 6638 MB]
  Finished Executing MultiApps                                                           [132004.69 s] [ 6638 MB]
 Solve Skipped!


*** ERROR ***
The following error occurred in the object "CD_angle", of type "PolynomialRegressionTrainer".

Reporter value storage/data:accumulate:CD_angle:value of size 8 does not match sampler size (75).

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[r3i0n0:mpi_rank_0][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0: No such file or directory (2)
[r3i0n4:mpi_rank_128][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x938c380). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r3i0n1:mpi_rank_32][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xd38abe70). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r3i0n25:mpi_rank_512][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x8943410). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r3i1n27:mpi_rank_1024][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0xa13dd40). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[r3i0n10:mpi_rank_256][mv2_ud_resend] UD reliability error. Exceeded max retries(513) in resending the message(0x8ed1a80). current retry timeout(us): 1000000. This Error may happen on clusters based on the InfiniBand topology and traffic patterns. Please try with increased timeout using MV2_UD_RETRY_TIMEOUT
[mpiexec@r3i0n0] HYDU_sock_write (../../../../src/pm/hydra/utils/sock/sock.c:303): write error (Bad file descriptor)
[mpiexec@r3i0n0] HYD_pmcd_pmiserv_send_signal (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy
[mpiexec@r3i0n0] ui_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream
[mpiexec@r3i0n0] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@r3i0n0] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event
[mpiexec@r3i0n0] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:340): process manager error waiting for completion

[grmnptr]

The last one includes an error in stochastic tools. But still, some ranks were still hanging. Have you seen anything like this @loganharbour , @milljm ?

[GiudGiud]

mvapich2/2.3.7-1-gcc-12.3.0

This does not work with the current timpi algorithm. You need to pass the timpi-sync command line parameters, or avoid mvapich (use openmpi or mpich) when compiling.

[harterj]

Man, I am bad at checking Github. So I switched and compiled Moose + Griffin with mpich on sawtooth, and that has given me stable behavior so far. Now just to wait on Sawtooth to come back up so I can test it more, but I used 80 nodes for a 3 day job and it worked just fine.

I was unsuccessful using openmpi on Lemhi, not sure how/if it functions on Sawtooth.