Esirkepov: Ensure Particle Does not Run out Dt

[ax3l]

Esirkepov: Ensure dt fits into cells with an assert

Re #4358
Follow-up to #4362

• Resolve comment by @roelof-groenewald in Electrostatic Diagnostics for j: Default to Direct Deposition #4362 (comment)

Can this be an AMREX_ASSERT_WITH_MESSAGE instead? The actual dt*PhysConst::c < min_dx is a stronger constraint than really necessary - would only be needed for particles traveling at the speed of light. I frequently use Esirkepov deposition with the hybrid solver (since it is faster than direct current deposition - not sure why - and less noisy).
The message might also be less rigid - something like "Time step could be too large for Esirkepov..."

[ax3l]

@roelof-groenewald in #4362 (comment)

Thanks for the reply. A runtime option to skip the assert would be fine. There is also currently a warning raised if a user chooses to use Esirkepov with the hybrid solver saying that if a particle gains too much speed a segfault will occur in the current deposition. It is not great for ensuring smooth code execution but it at least gives the user a starting place to look if they find that their simulation crashes.

An alternative option I have seen used is to simply limit particle speeds - using the same expression you have to calculate the maximum allowable speed and if a particle is pushed so that it would exceed that speed its speed is set equal to the maximal allowed speed. This obviously violates energy conservation but it prevents the code from crashing. I guess a different strategy could be to set the speed of light to the maximum supported speed in which case the speed limit will be naturally applied through the relativistic corrections... All of that is just for interest sake, I'm not suggesting we do that.

Regarding Esirkepov being faster than direct deposition. I see that with few GPU runs but with many particles, using the hybrid solver. For example, a simulation using 2 Perlmutter GPUs with 100 x 100 x 184 grid size and 400M particles achieves ~1.4 steps per second with Esirkepov but only ~1.15 steps per second with direct deposition. I'll add this to the dev meeting agenda for next week.

[ax3l]

Do not check on ElectromagneticSolverAlgo::HybridPIC / algo.maxwell_solver = hybrid, ... https://warpx.readthedocs.io/en/latest/usage/parameters.html#maxwell-solver

Technically, only for EM sims with Yee, CKC, PSATD, ...