Running the melt pool 3D example

[alimostafavi24]

Hello,

When I run the melt pool 3D case in MALAMUTE examples, I got the following error on HPC using 92 CPUs. Any suggestion to tackle this issue is highly appreciated.

Thanks,
Ali

�[36mreinit0: �[39m
�[36mreinit0: �[39mCommand Line Argument(s):
�[36mreinit0: �[39m  --allow-unused
�[36mreinit0: �[39m
�[36mreinit0: �[39mParallelism:
�[36mreinit0: �[39m  Num Processors:          92
�[36mreinit0: �[39m  Num Threads:             1
�[36mreinit0: �[39m
�[36mreinit0: �[39mMesh: 
�[36mreinit0: �[39m  Parallel Type:           replicated
�[36mreinit0: �[39m  Mesh Dimension:          3
�[36mreinit0: �[39m  Spatial Dimension:       3
�[36mreinit0: �[39m  Nodes:                   
�[36mreinit0: �[39m    Total:                 132651
�[36mreinit0: �[39m    Local:                 1823
�[36mreinit0: �[39m    Min/Max/Avg:           974/1823/1441
�[36mreinit0: �[39m  Elems:                   
�[36mreinit0: �[39m    Total:                 125000
�[36mreinit0: �[39m    Local:                 1383
�[36mreinit0: �[39m    Min/Max/Avg:           1319/1399/1358
�[36mreinit0: �[39m  Num Subdomains:          1
�[36mreinit0: �[39m  Num Partitions:          92
�[36mreinit0: �[39m  Partitioner:             metis
�[36mreinit0: �[39m
�[36mreinit0: �[39mNonlinear System:
�[36mreinit0: �[39m  Num DOFs:                530604
�[36mreinit0: �[39m  Num Local DOFs:          7292
�[36mreinit0: �[39m  Variables:               "ls" "grad_ls" 
�[36mreinit0: �[39m  Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC" 
�[36mreinit0: �[39m  Approximation Orders:    "FIRST" "FIRST" 
�[36mreinit0: �[39m
�[36mreinit0: �[39mAuxiliary System:
�[36mreinit0: �[39m  Num DOFs:                257651
�[36mreinit0: �[39m  Num Local DOFs:          3206
�[36mreinit0: �[39m  Variables:               "ls_0" "box" 
�[36mreinit0: �[39m  Finite Element Types:    "LAGRANGE" "MONOMIAL" 
�[36mreinit0: �[39m  Approximation Orders:    "FIRST" "CONSTANT" 
�[36mreinit0: �[39m
�[36mreinit0: �[39mExecution Information:
�[36mreinit0: �[39m  Executioner:             Transient
�[36mreinit0: �[39m  TimeStepper:             ConstantDT
�[36mreinit0: �[39m  TimeIntegrator(s):       ImplicitEuler
�[36mreinit0: �[39m  Solver Mode:             NEWTON
�[36mreinit0: �[39m  PETSc Preconditioner:    lu mat_superlu_dist_fact: SamePattern  mat_superlu_dist_replacetinypivot: true 
�[36mreinit0: �[39m  MOOSE Preconditioner:    SMP (auto)
�[36mreinit0: �[39m
�[36mreinit0: �[39m�[31mLEGACY MODES ENABLED:�[39m
�[36mreinit0: �[39m This application uses the legacy initial residual evaluation behavior. The legacy behavior performs an often times redundant residual evaluation before the solution modifying objects are executed prior to the initial (0th nonlinear iteration) residual evaluation. The new behavior skips that redundant residual evaluation unless the parameter Executioner/use_pre_smo_residual is set to true. To remove this message and enable the new behavior, set the parameter 'use_legacy_initial_residual_evaluation_behavior' to false in *App.C. Some tests that rely on the side effects of the legacy behavior may fail/diff and should be re-golded.
�[36mreinit0: �[39m�[39m
�[36mreinit0: �[39mCurrently Performing Initial Setup�[36mreinit0: �[39m
�[36mreinit0: �[39m  Performing Initial Adaptivity�[36mreinit0: �[39m
�[36mreinit0: �[39m    Handling Mesh Changes                                                       [�[33m 10.34 s�[39m] [�[33m 3064 MB�[39m]
�[36mreinit0: �[39m    Adapting Mesh....                                                           [�[33m 33.41 s�[39m] [�[33m 3677 MB�[39m]
�[36mreinit0: �[39m    Handling Mesh Changes...........                                            [�[33m 71.38 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39m  Finished Performing Initial Adaptivity                                        [�[33m123.94 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39m  Building SemiLocalElemMap....                                                 [�[33m 32.70 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39mFinished Performing Initial Setup                                               [�[33m157.01 s�[39m] [�[33m 4298 MB�[39m]
    Finished Initializing MultiApps                                                      [�[33m157.01 s�[39m] [�[33m 4298 MB�[39m]
    Building SemiLocalElemMap....                                                        [�[33m 32.53 s�[39m] [�[33m 4298 MB�[39m]
  Still Performing Initial Setup..
  Finished Performing Initial Setup                                                      [�[33m356.38 s�[39m] [�[33m 4298 MB�[39m]

Time Step 0, time = 0
  Currently Outputting
    Outputting Step.                                                                     [�[33m 15.83 s�[39m] [�[33m 4411 MB�[39m]
  Finished Outputting                                                                    [�[33m 15.84 s�[39m] [�[33m 4411 MB�[39m]

Time Step 1, time = 0.0001, dt = 0.0001
 * Nonlinear |R| = �[32m1.617244e-07�[39m (Before preset BCs, predictors, correctors, and constraints)

Performing automatic scaling calculation


    Computing Automatic Scaling
      Computing Jacobian.........                                                        [�[33m 52.11 s�[39m] [�[33m 4413 MB�[39m]
    Finished Computing Automatic Scaling                                                 [�[33m 52.48 s�[39m] [�[33m 4413 MB�[39m]
 0 Nonlinear |R| = �[32m3.988573e-08�[39m
  Finished Solving                                                                       [�[33m 61.02 s�[39m] [�[33m 4413 MB�[39m]
�[32m Solve Converged!�[39m
Still Executing.Solving Reinitialization problem.
�[36mreinit0: �[39m
�[36mreinit0: �[39mTime Step 0, time = 0
�[36mreinit0: �[39m
�[36mreinit0: �[39mTime Step 1, time = 1e-05, dt = 1e-05
�[36mreinit0: �[39m * Nonlinear |R| = �[32m1.738155e-07�[39m (Before preset BCs, predictors, correctors, and constraints)
�[36mreinit0: �[39m 0 Nonlinear |R| = �[32m1.738155e-07�[39m
�[36mreinit0: �[39m
�[36mreinit0: �[39m  Computing Jacobian                                                            [�[33m 10.08 s�[39m] [�[33m 4489 MB�[39m]
[r3u05n1:3207054:0:3207054]      cma_ep.c:85   process_vm_readv(pid=3207053 {0x14a99e9efa38,524288}-->{0x1490ca5f9a38,524288}) returned -1: No such process
[r3u05n1:3207047:0:3207047]      cma_ep.c:85   process_vm_readv(pid=3207048 {0x14cbbcdf5030,524288}-->{0x14cd6db0a030,524288}) returned -1: No such process
[r3u05n1:3207052:0:3207052]      cma_ep.c:85   process_vm_readv(pid=3207050 {0x15265252d140,524288}-->{0x1549d2c8a140,524288}) returned -1: No such process
--------------------------------------------------------------------------
prterun noticed that process rank 69 with PID 3207048 on node r3u05n1 exited on
signal 9 (Killed).
--------------------------------------------------------------------------
Apr 20 15:31:37.599787 3206920 slurmstepd   0x15041958cfc0: error: Detected 4 oom_kill events in StepId=12751509.batch. Some of the step tasks have been OOM Killed.

[alimostafavi24]

Regarding my previous comment, the following preconditioner is too expensive, and could explain why I got out of memory error.

  petsc_options_iname = '-pc_type -pc_factor_shift_type -pc_factor_mat_solver_package -ksp_type'
  petsc_options_value = 'lu NONZERO superlu_dist preonly'

Are there any alternatives that known to work well for this problem? for example I tried the following and faced this error:

  petsc_options_iname = '-pc_type -pc_asm_overlap -sub_pc_type -sub_pc_factor_levels  -ksp_gmres_restart'
  petsc_options_value = 'bjacobi      2               ilu          3       200'

 0 Nonlinear |R| = �[32m1.476174e-07�[39m
    Computing Jacobian.                                                                  [�[33m 10.51 s�[39m] [�[33m  608 MB�[39m]
    Linear solve did not converge due to DIVERGED_PC_FAILED iterations 0
                   PC failed due to SUBPC_ERROR 
  Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0
�[31m Solve Did NOT Converge!�[39m