Improving Performance of AD Material Models in MOOSE with Many Parameters

[vikramroybarc]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi Moose Team,

I converted my original (non-AD) material model to an AD version by replacing types like Real and RankTwoTensor with ADReal and ADRankTwoTensor, and by using the appropriate getters such as getADMaterialProperty.

However, the AD implementation is significantly slower—around 10-15× compared to the original non-AD version. I have automatic_scaling = true enabled in both cases.

The material model also includes around 20–25 input parameters (see: https://github.com/apatra6/rhocp/blob/main/src/materials/DDJ2StressUpdate.C). I’ve also read that AD performance can degrade with a large number of parameters (https://doi.org/10.1080/00295450.2020.1838877).

Would it help to convert variables that do not contribute to the Jacobian and constant variables from ADReal back to Real to improve performance?

More generally, could anyone suggest strategies to speed up the AD version? I need it for multiapp phase-field fracture simulations, where the non-AD version tends to fail, likely due to Jacobian-related issues.

Thanks,
Vikram Roy

[GiudGiud]

Hello

Would it help to convert variables that do not contribute to the Jacobian and constant variables from ADReal back to Real to improve performance?

Yes it's possible it will help. The best way to gather more information on this would be to profile before optimizing:
https://mooseframework.inl.gov/application_development/profiling.html

You will need to re-compile in oprof mode

likely due to Jacobian-related issues.

You can investigate that by using finite difference preconditioning btw.
https://mooseframework.inl.gov/application_usage/failed_solves.html

[vikramroybarc]

Hi @GiudGiud,

You can investigate that by using finite difference preconditioning btw.
https://mooseframework.inl.gov/application_usage/failed_solves.html

Thanks for the tip—I tried using finite difference preconditioning, and it successfully resolved the convergence failures I was seeing.
Moving forward, I’m curious about the performance trade-offs. In terms of solution speed and robustness, is it generally more efficient to use FD-based preconditioning or to implement an AD-based material model?
Thanks!

[vikramroybarc]

Hi @GiudGiud,

How can I output the Jacobian for the AD and FD cases?

I’m working on phase-field fracture simulations with plasticity, so the Jacobian of my material model includes a plastic contribution. With this Jacobian, the solver performs well up to about a 4–5% load drop from the peak. However, once crack growth initiates and further load drops occur, the solution process fails.

The typical stress–strain response from my simulation is shown in the figure. Using my current Jacobian, I can solve up to point C, but I’m unable to capture the subsequent load-drop behavior. In contrast, this part of the response can be successfully captured when using AD or FD.

[GiudGiud]

How can I output the Jacobian for the AD and FD cases?

You can use -ksp_view_pmat to see the preconditioning matrix, which for Newton (or full=true in PJFNK) preconditioning is the Jacobian

See commands there
https://petsc.org/release/manualpages/KSP/KSPSolve/

The typical stress–strain response from my simulation is shown in the figure. Using my current Jacobian, I can solve up to point C, but I’m unable to capture the subsequent load-drop behavior. In contrast, this part of the response can be successfully captured when using AD or FD.

and it fails without AD for both Newton and PJFNK?
what petsc options are you using?

[lindsayad]

Were you seeing faster performance with FD than AD? That would be surprising and emphasize the need for profiling. If you are seeing that, then that would be impetus for bringing back older derivative container implementations that don't compute sparsity patterns on-the-fly.

(or full=true in PJFNK) preconditioning is the Jacobian

If the Jacobian is linear and the floating point arithmetic is well-behaved, then this is true. But generally it is not true and this was a major reason for the creation of AD

[vikramroybarc]

Hi @lindsayad & @GiudGiud,

Thank you for suggesting the PJFNK solver — I’m now able to solve the problem without failures, and it’s currently the fastest approach, even without using an FD-based preconditioner.

Here are the PETSc options I’m using:

petsc_options = '-snes_ksp_ew'
petsc_options_iname = '-pc_type -pc_factor_mat_solver_package'
petsc_options_value = 'lu superlu_dist'
automatic_scaling = true

Do these settings look appropriate, or would you recommend any modifications or corrections?

Additionally, I’m observing faster performance with AD compared to FD.

Thanks again for your guidance.

[GiudGiud]

ok glad there is improvement.

petsc_options_iname = '-pc_type -pc_factor_mat_solver_package'
petsc_options_value = 'lu superlu_dist'

this method does not scale to large problems. How many degrees of freedom are there in the problem?

Additionally, I’m observing faster performance with AD compared to FD.

ok this is as expected. The best performance with PJFNK can be achieved without AD. If you have a non-AD version of all your materials and kernels you can try it.

[vikramroybarc]

How many degrees of freedom are there in the problem?

In my case I have around 7000 nodes, with each having 3 dof.

this method does not scale to large problems.

Which PETSc options are recommended for handling large-scale problems with a high number of nodes?

[lindsayad]

PJFNK is faster than Newton when using AD? That suggests to me that not the Jacobian is not perfect. How many nonlinear iterations with Newton vs. PJFNK?

[vikramroybarc]

Hi @lindsayad,

I ran more thorough tests on the code. For the 2D case, AD with Newton is about twice as fast as the non-AD approach with PJFNK.

However, for the 3D case, the non-AD method with PJFNK is roughly three times faster than Newton with AD.

The 3D setup includes only 27 Hex8 elements, and I’m using the following PETSc options:

petsc_options_iname = '-pc_type -pc_factor_mat_solver_package'
petsc_options_value = 'lu superlu_dist'

How many nonlinear iterations with Newton vs. PJFNK?

In both cases 20 NL iterations

[lindsayad]

How many linear iterations do you see on average when using PJFNK?

[vikramroybarc]

max 3-4 iterations.

[lindsayad]

I ran more thorough tests on the code. For the 2D case, AD with Newton is about twice as fast as the non-AD approach with PJFNK.

However, for the 3D case, the non-AD method with PJFNK is roughly three times faster than Newton with AD.

This makes sense. Sparsity union calculations with AD will be significantly more expensive in 3D than 2D. Especially given that nonlinear finite volume with its big, complicated stencils is being conceptually deprecated in favor of the small stencil linear finite volume which doesn't use AD, it may make increasing sense to support a local AD seeding scheme again which would avoid these sparsity union calculations. I've opened #32883 in that vein. Unfortunately, I don't imagine anyone on the MOOSE team working on this any time soon