[WIP] Fix: RedistributeCPU for SoA

[ax3l]

Summary

This is a first draft and needs to be generalized.

Additional background

Follow-up to #2878.
ImpactX w/ OMP now: ECP-WarpX/impactx#348

• not crashing with space charge anymore
• results still off expectations: we loose particles now.

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[ax3l]

What is this for a pure SoA particle?
Does the later

aos.insert(aos.end(), aos_tmp[i].begin(), aos_tmp[i].end());
aos_tmp[i].erase(aos_tmp[i].begin(), aos_tmp[i].end());

work as we expect?

Let's add more inline comments for the individual code blocks, so we understand the in-transit data structure and logic each of the many loops/copies/moves executes :)

Not clear to me if in-transit it is still an SoA and when what is overwritten. I see that later AoS attributes are not communicated correctly in ImpactX (e.g., momenta) but cannot say why yet.

[ax3l]

Underlying bug fixed via #3296.

We keep this open since we want to check if sections like

tmp_local[pld.m_lev][index][thread_num].push_back(p);

are well-defined/sensible for pure SoA particles or if we need additional branching there.

[ax3l]

Continued in #3299