Fix bug with MLMG solver, always pass ghost cells to SumBoundary

[EZoni]

Following up on a bug raised by @aeriforme related to the convergence of the MLMG solver, and based on a comment posted on Slack by @WeiqunZhang (channel #general, May 30, 2023),

There are three versions of SumBoundary's in ablastr/utils/Communication.cpp. Their differences are about the number of ghost cells included in the operation. I think we should remove two of them and force us to explicitly think and state the number of source and destination ghost cells.

let's try to keep only the most general version of SumBoundary (the last one among the three shown below)

WarpX/Source/ablastr/utils/Communication.H

Lines 85 to 102 in f30799b

	void SumBoundary (amrex::MultiFab &mf,
	bool do_single_precision_comms,
	const amrex::Periodicity &period = amrex::Periodicity::NonPeriodic());
	void SumBoundary (amrex::MultiFab &mf,
	int start_comp,
	int num_comps, amrex::IntVect ng,
	bool do_single_precision_comms,
	const amrex::Periodicity &period = amrex::Periodicity::NonPeriodic());
	void
	SumBoundary (amrex::MultiFab &mf,
	int start_comp,
	int num_comps,
	amrex::IntVect src_ng,
	amrex::IntVect dst_ng,
	bool do_single_precision_comms,
	const amrex::Periodicity &period = amrex::Periodicity::NonPeriodic());

and request that developers pass the number of source and destination ghost cells explicitly, each time.

@aeriforme @WeiqunZhang
Do you have a workflow to test this PR on the MLMG convergence bug? Did you expect SumBoundary to be called in many other parts of the code? I found only a few instances related to the deposition of the charge density rho, and I'm not sure which one(s) of these should use rho->nGrowVect() instead of amrex::IntVect(0) as dst_ng in order to fix the MLMG convergence bug. Do you know?

@ax3l @RemiLehe
Please feel free to comment or suggest alternative solutions.
Here I'm trying to implement @WeiqunZhang's idea above.

[ax3l]

Something to double-check in ImpactX, too?
https://github.com/ECP-WarpX/impactx/tree/development/src/particles/spacecharge

[EZoni]

@lijiangdong1999
This is the PR mentioned on Gitter, which you can try to run on your AWAKE input setup.

[lijiangdong1999]

@lijiangdong1999 This is the PR mentioned on Gitter, which you can try to run on your AWAKE input setup.

Thanks for your reply. But I use spack to install warpx, I can't find the "Source/Particles/WarpXParticleContainer.cpp".

[EZoni]

Add inline comments on possible performance optimization.

[EZoni]

For future reference, on this input deck, the MLMG solver fails on the development branch and converges on this branch.