Error: SoAParticle::NextID() -- too many particles when continuously injecting electrons in RZ HET simulation

[Simone-Di-Fede]

Hello,

I am simulating a Hall Effect Thruster (HET) in RZ geometry, similar to the setup discussed in Discussion #6486.

In my simulation, I am continuously injecting new electrons at the cathode and handling secondary electron emission using the electron_wrapper.add_particles method, drawing inspiration from this implementation (https://github.com/archermarx/warpx-hall/blob/main/iepc_2025/hallx.py).

The code runs as expected. But I need to run the code for a long "physical" period and after some time it crashes with the following error:
amrex::Abort::0::SoAParticle<NArrayReal, NArrayInt>::NextID() -- too many particles !!!

I suspect this is happening because every time electron_wrapper.add_particles is called, new particle IDs are generated, eventually exceeding the maximum allowed limit.

I have a couple of questions regarding this issue:

Is there a way to reset or recycle the particle IDs during runtime to avoid hitting this limit?

Is it possible to reset the particle IDs when restarting a simulation from a checkpoint?

How big is this particle IDs limit? Is it possible to increase it?

Any guidance or suggestions would be greatly appreciated. Thank you! I am compiling the code in double precision.

[dpgrote]

@WeiqunZhang This is a question for you. I notice in AMReX Src/Particle/AMReX_Particle.H has LastParticleID set so 2^39 - 3. Is there a reason that it is not 2^63 - 3?

[Simone-Di-Fede]

Thank you for the information. The limit of 2^39 is indeed consistent with the timestep at which I encounter the error, considering the setup I described previously. This confirms that the crash is definitely caused by the continuous generation of new particle IDs due to electron injection and secondary electron emission.If the ID limit could be increased to 2^63 it would obviously avoid this issue entirely. Could you please let me know if it is possible to configure or extend this limit?

[WeiqunZhang]

For an amrex particle, a 64 bits integer is used for ID and CPU, with 40 bits for ID and 24 bits for CPU. For the ID part, 1 bit is used as a flag for valid or not. Thus, there are 39 bits left for the valid IDs.

[ax3l]

Note: if you inject your particles from more than one MPI rank, or manually tinker with the cpu id in non-MPI scenarios, then you can use the 24 bits of the CPU as well.

So in practice, you could use up to 63 bits (2^63) if this is a limitation for you.

We understand that the problem can appear when injecting new particles over a long time always on the same (or same few, e.g., boundary) MPI ranks. Creating 2^39 particles per MPI rank and simulation does not need to stay a fundamental limitation.

[Simone-Di-Fede]

Thanks everyone for the clear explanation.

For the time being, I've implemented a workaround using class FromFileDistribution(picmistandard.PICMI_FromFileDistribution):. It's not the most practical solution, but it generally allows me to move forward with the code development. Crucially, unlike a standard restart, using FromFileDistribution "clears the particle IDs", which temporarily solves the overflow issue.

I am still evaluating whether to proceed with modifying the source code. Since I run the code on a single GPU, I could indeed modify the C++ to manually tinker with the CPU ID and push the limit up to 63 bits as suggested in non-MPI scenarios. However, at the same time, it is crucial for me to easily maintain updates with the original repository, which obviously breaks (or becomes a headache to manage) if I heavily modify the source code locally.

I will evaluate the best path forward. Thanks again for the support!