MLMG failed for UserDefined EB

[JayStandridge]

Hi all,

I am trying to simulate Sandia flame D. I am using the UserDefined EB option to specify the geometry. However, when running I encounter the same MLMG error encountered previously in these discussions. As recommended then and in the documentation, I have set the verbosity levels to three. The issue is still not clear to me and I am not sure what my next steps ought to be.
To begin, I have started with a 2D flow. I am sufficiently resolved in the coarse mesh to make sure the length of the coarse cells are around the same length as the width of the tubes in the flame D geometry. Here is my mesh:

I have changed the values of the tube width so that they align with the fine cells in hopes that it would solve this EB error. Here is the output of the code when running:

MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing SUNDIALS with 1 threads...
SUNDIALS initialized.
AMReX (23.07-4-g9fc635426cae) initialized
Successfully read inputs file ... 

 ================= Build infos =================
 PeleLMeX    git hash: a7364cb-dirty
 AMReX       git hash: 23.07-4-g9fc6354-dirty
 PelePhysics git hash: v23.03-50-g8fd8071-dirty
 AMReX-Hydro git hash: f24f4a6
 ===============================================

 ==============================================================================
 State components 
 ==============================================================================
 Velocity X: 0, Velocity Y: 1 
 Density: 2
 First species: 3
 Enthalpy: 24
 Temperature: 25
 thermo. pressure: 26
 => Total number of state variables: 27
 ==============================================================================

 Initializing data for 3 levels 
 Initialization of Transport ... 
    Using LES in transport with Sc = 0.7 and Pr = 0.7
 Initialization of chemical reactor ... 
Creating ReactorBase instance: ReactorCvode
Initializing CVODE:
 Making new level 0 from scratch
 Making new level 1 from scratch
 with 266240 cells,4 boxes, over 1.586914062% of the domain 
 Making new level 2 from scratch
 with 396288 cells,14 boxes, over 0.5905151367% of the domain 
 Making new level 1 from scratch
 with 332800 cells,4 boxes, over 1.983642578% of the domain 
 Making new level 2 from scratch
 with 528384 cells,14 boxes, over 0.7873535156% of the domain 
 Resetting fine-covered cells mask 
 Update chemistry typical values 
 ==============================================================================
 Typical values: 
	Velocity: 1e-05 0 
	Density:  1.146769749
	Temp:     294
	H:        -211425.31
	Y_H2:     0
	Y_H:      0
	Y_O:      0
	Y_O2:     0.2226345
	Y_OH:     0
	Y_H2O:    0
	Y_HO2:    0
	Y_CH2:    0
	Y_CH2(S): 0
	Y_CH3:    0
	Y_CH4:    0.0445269
	Y_CO:     0
	Y_CO2:    0
	Y_HCO:    0
	Y_CH2O:   0
	Y_CH3O:   0
	Y_C2H4:   0
	Y_C2H5:   0
	Y_C2H6:   0
	Y_N2:     0.7328386
	Y_AR:     0
 ==============================================================================
 Est. time step - Conv: 0.9155273437, divu: 1e+20
 Initial dt: 0.0009155273437

 Writing plotfile: plt00000

 Writting checkpoint file: chk00000
 ==============================================================================

 ====================   NEW TIME STEP   ==================== 
 Est. time step - Conv: 0.9155273437, divu: 1e+20
 STEP [0] - Time: 0, dt 0.0009155273437
   Computing Turbulent Viscosity with LES model: Smagorinsky for time 0
   SDC iter [1] 
MLMG: # of AMR levels: 3
      # of MG levels on the coarsest AMR level: 2
MLMG: Initial rhs               = 5.752584538
MLMG: Initial residual (resid0) = 5.752584538
MLMG: Bottom solve failed.
MLMG: Iteration   1 Fine resid/bnorm = 0.02745314068
MLMG: Bottom solve failed.
MLMG: Iteration   2 Fine resid/bnorm = 0.001496304308
MLMG: Bottom solve failed.
MLMG: Iteration   3 Fine resid/bnorm = 0.0009348682497
MLMG: Bottom solve failed.
MLMG: Iteration   4 Fine resid/bnorm = 0.0006973341725
MLMG: Bottom solve failed.
MLMG: Iteration   5 Fine resid/bnorm = 0.0005569493601
MLMG: Bottom solve failed.

 ...

MLMG: Iteration 198 Fine resid/bnorm = 4.585708101e-05
MLMG: Bottom solve failed.
MLMG: Iteration 199 Fine resid/bnorm = 4.574153318e-05
MLMG: Bottom solve failed.
MLMG: Iteration 200 Fine resid/bnorm = 4.562685442e-05
MLMG: Failed to converge after 200 iterations. resid, resid/bnorm = 0.0002624723373, 4.562685442e-05
amrex::Abort::0::MLMG failed !!!

The solver fails to reach the default 1e-11 or the maximum recommended limit of 5e-10, which would indicate my problem is not properly set up. Where should I look to fix this issue?

Thank you in advance for your assistance.

[baperry2]

This sort of thin EB surface will tend to cause problems for the multigrid linear solvers from AMReX used by PeleLMeX. If the geometry can't be coarsened because the body that must be resolved is too thin, the solvers will not work well. The simplest and best "fix" would be to adjust your computational domain to be flush with the nozzle exit plane of the burner, and thereby avoid the need for EB altogether, if you can do so.

Otherwise, you can try some things to help the linear solvers along - better initial velocity field, more iterations, gradually build refinement, etc - but getting the solvers to converge for this case might be a struggle. There's an undocumented ability to interface with HYPRE to circumvent this, you can see Exec/Production/ChallengeProblem for an example, but it could take a bit of effort to get it set up properly for your problem. This is a known issue, and there's a plan to add additional solvers that make PeleLMeX more robust with this sort of geometry, but the timing is uncertain.

[hsitaram]

As @baperry2 suggests, hypre could potentially help. add USE_HYPRE=TRUE in the makefile and add nodal_proj.mg_max_coarsening_level = 0 to get it going. It can be quite a bit slower than geometric multigrid.

[ZhaofanZhu]

Hi, Mr.Bruce @baperry2 , I'm trying my EB case (#327 ) with HYPRE. I refer to the settings in Exec/Production/ChallengeProblem.

Using HYPRE, my case with periodic calculations in both directions (#327 ) did run. However, it overflows my server's memory after running a certain number of steps. Even if I restart where I left off, it still causes memory overflow after a certain number of steps.

 ====================   NEW TIME STEP   ==================== 
 STEP [337] - Time: 2.74501532e-05, dt 1e-07
   SDC iter [1] 
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=6471259.0. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: nid001093: tasks 0-127: Out Of Memory
srun: launch/slurm: _step_signal: Terminating StepId=6471259.0

I've tried to figure out why, but there doesn't seem to be a proper user guide. Do you have any advice? Thank you in advance!

This is my input settings:

input

#---------------------- DOMAIN DEFINITION ------------------------
geometry.is_periodic = 1 0 1 # For each dir, 0: non-perio, 1: periodic
geometry.coord_sys = 0 # 0 => cart, 1 => RZ
geometry.prob_lo = 0 0 0 # x_lo y_lo (z_lo)
geometry.prob_hi = 0.002 0.002 0.001 # x_hi y_hi (z_hi)

#---------------------- BC FLAGS ---------------------------------
peleLM.lo_bc = Interior NoSlipWallAdiab Interior # bc in x_lo y_lo (z_lo)
peleLM.hi_bc = Interior Outflow Interior # bc in x_hi y_hi (z_hi)

#---------------------- AMR CONTROL ------------------------------
amr.n_cell = 128 128 64 # Level 0 number of cells in each direction
amr.v = 1 # AMR verbose
amr.max_level = 0 # maximum level number allowed
amr.ref_ratio = 2 2 2 2 # refinement ratio
amr.regrid_int = 5 # how often to regrid
amr.n_error_buf = 2 2 2 2 # number of buffer cells in error est
amr.grid_eff = 0.7 # what constitutes an efficient grid
amr.blocking_factor = 16 # block factor in grid generation (min box size)
amr.max_grid_size = 32 # max box size

#---------------------- Problem ----------------------------------
prob.T_mean = 298.0
prob.P_mean = 101325.0
prob.T_hot = 1800
prob.T_wall = 298.0
prob.Y_fuel = 0.02294
prob.Y_oxid = 0.22756
prob.flame_loc = 0.001

prob.b_H2O = 0.20210
prob.b_O2 = 0.04466
prob.b_H2 = 1.18e-4
prob.b_OH = 3.34e-3
prob.b_H = 7.51e-6
prob.b_O = 2.74e-4

#---------------------- PeleLM CONTROL ---------------------------
peleLM.v = 1 # PeleLMeX verbose
peleLM.use_wbar = 1 # Include Wbar term in species diffusion fluxes
peleLM.sdc_iterMax = 2 # Number of SDC iterations
peleLM.num_init_iter = 3 # Number of initial iterations
peleLM.floor_species = 0 # Enforce species positivity (non-conservative !)

#---------------------- Temporal CONTROL -------------------------
peleLM.do_temporals = 1 # Turn temporals ON/OFF
peleLM.temporal_int = 10 # Frequency of temporals
peleLM.do_extremas = 1 # Compute state extremas
peleLM.do_mass_balance = 1 # Compute mass balance
peleLM.do_species_balance = 1 # Compute species balance

#---------------------- Time Stepping CONTROL --------------------
amr.max_step = 5000 # Maximum number of time steps
amr.stop_time = 0.2 # final simulation physical time
#amr.cfl = 0.2 # CFL number for hyperbolic system
amr.fixed_dt = 0.2e-7
amr.dt_shrink = 0.001 # Scale back initial timestep
amr.dt_change_max = 1.1 # Maximum dt increase btw successive steps

#---------------------- IO CONTROL -------------------------------
amr.restart = chk02700 # Restart checkpoint file
amr.check_int = 100 # Frequency of checkpoint output
amr.plot_int = 50 # Frequency of pltfile output
amr.derive_plot_vars = avg_pressure mag_vort mass_fractions

#---------------------- Reactor CONTROL --------------------------
peleLM.chem_integrator = "ReactorCvode"
peleLM.use_typ_vals_chem = 1 # Use species/temp typical values in CVODE
ode.rtol = 1.0e-6 # Relative tolerance of the chemical solve
ode.atol = 1.0e-5 # Absolute tolerance factor applied on typical values
ode.clean_init_massfrac = 0 # Reaction computed as an increment
cvode.solve_type = denseAJ_direct # CVODE Linear solve type (for Newton direction)
cvode.max_order = 4 # CVODE max BDF order.

#---------------------- EB SETUP ---------------------------------
eb2.geom_type = plane
eb2.plane_point = 0 0.0002 0
eb2.plane_normal = 0 -0.02 0
eb2.plane_has_fluid_inside = 0
eb2.small_volfrac = 1.e-4
peleLM.isothermal_EB = 1

#---------------------- Refinement CONTROL------------------------
amr.refinement_indicators = gradT
amr.gradT.max_level = 3
amr.gradT.adjacent_difference_greater = 100
amr.gradT.field_name = temp

#---------------------- Linear solver CONTROL --------------------
nodal_proj.verbose = 0 # Nodal projection verbose
mac_proj.verbose = 0 # MAC projection verbose

nodal_proj.verbose = 1
nodal_proj.mg_max_coarsening_level = 0
nodal_proj.rtol = 1.0e-11
nodal_proj.atol = 1.0e-11
nodal_proj.bottom_solver = "hypre"
nodal_proj.hypre_namespace = nodal_proj.hypre

nodal_proj.hypre.verbose = 1
nodal_proj.hypre.hypre_solver = GMRES
nodal_proj.hypre.hypre_preconditioner = BoomerAMG
nodal_proj.hypre.num_krylov = 200
nodal_proj.hypre.max_iterations = 200

nodal_proj.hypre.bamg_verbose = 0
nodal_proj.hypre.bamg_coarsen_type = 8
nodal_proj.hypre.bamg_interp_type = 6
nodal_proj.hypre.bamg_relax_type = 11
nodal_proj.hypre.bamg_num_sweeps = 2
nodal_proj.hypre.bamg_cycle_type = 1
nodal_proj.hypre.bamg_relax_order = 0
nodal_proj.hypre.bamg_keep_transpose = 1
nodal_proj.hypre.bamg_trunc_factor = 0.44351885396256086
nodal_proj.hypre.bamg_strong_threshold = 0.30015419030906665
nodal_proj.hypre.bamg_pmax_elmts = 4

nodal_proj.hypre.bamg_agg_num_levels = 3
nodal_proj.hypre.bamg_agg_interp_type = 5
nodal_proj.hypre.bamg_agg_pmax_elmts = 5

nodal_proj.hypre.bamg_smooth_type = 5
nodal_proj.hypre.bamg_smooth_num_sweeps = 1
nodal_proj.hypre.bamg_smooth_num_levels = 1
nodal_proj.hypre.bamg_ilu_type = 0
nodal_proj.hypre.bamg_ilu_level = 0
nodal_proj.hypre.bamg_ilu_max_iter = 1
nodal_proj.hypre.bamg_ilu_reordering_type = 0
nodal_proj.hypre.bamg_ilu_tri_solve = 0
nodal_proj.hypre.bamg_ilu_lower_jacobi_iters = 13
nodal_proj.hypre.bamg_ilu_upper_jacobi_iters = 5

fabarray.mfiter_tile_size = 1024 1024 1024

amrex.abort_on_out_of_gpu_memory = 1
<\details>

[baperry2]

I have not used the HYPRE capability so I don't know what this could be. @hsitaram or @nickwimer did you ever run into something like this?

[ZhaofanZhu]

Thank you, Mr.Bruce. Hi, Mr. @hsitaram and @nickwimer , have you ever encountered such a problem?

[hsitaram]

Hi @ZhaofanZhu - I have not encountered OOMs with hypre. Can you reduce the nodal_proj.hypre.num_krylov to 100? could help a bit. I also suggest looking at the settings in the EB_EnclosedFlame test in Exec which has slightly different hypre settings.