Calculate particles min-max and corresponding overheads

[Change72]

Hi, I met several questions in my WarpX simulation:

1. I try to generate a dataset with particles min-max in metadata. I found there is a setting 'ParticleExtrema' and also an example inputs: inputs_3d and inputs_3d_multiple_particles

Since I want to use ADIOS 2 as the backend, the only thing I changed is:

diag1.format = openpmd 
diag1.openpmd_backend = bp 
diag1.openpmd_encoding = g

After generation, I use bpls -l to check the min-max value, but it shows all 0.

2. in bpls --help, it shows:

 --long      | -l           Print values of all scalars and attributes and
                               min/max values of arrays (no overhead to get them!)

My question is whether there will be some kind of overheads in collecting min/max if WarpX using Adios 2 backend, like 'ParticleExtrema'? If so, does such a cost is negligible or significant?

3. in my previous issues Generating dataset on Summit: GPU last error detected #4438 @ax3l mention Another small unrelated detail: your job seems to use 12 GPUs (2 nodes?) but your setup only has 8 grid blocks, so 4 GPUs will be unused right now. For now, I try to control the grids number by Recommend configuration for large scale particles #4591 setting amr.max_grid_size and amr.blocking_factor explicitly to the same numbers. Does this mean if I want to achieve near-optimal performance with 4 nodes and each of them has 2 GPUS, I need at least set it to 8 grids or say 8x grids, like 16 or 24?

4. what is the relationship of AMReX and ADIOS2 in WarpX? As far as I know, AMReX output format is plotfile and ADIOS2 is bp files. It seems they are two different I/O formats. However, in the last page of ADIOS2 talk, it tells Codes such as the WarpX code, which uses AMReX can take advantage of ADIOS-BP4 for “fast” writing. So I am a little puzzled now.

5. I use this input_3d to do a simulation, and then I read it by ADIOS engine. However, I found an interesting thing is the total particle number is matched with bpls -l but the block number is 128. I cannot find any 128 in input files and output files. Also, I try another large dataset, and get a weird number as well. I use the following to read block number:

    bpIO = adios.DeclareIO("ReadBP");
    bpIO.SetParameter("Threads", std::to_string(nThreads));
    bpReader = bpIO.Open(bpFileName, adios2::Mode::Read);

    adios2::Variable<double> x_meta_info = bpIO.InquireVariable<double>(key + "x");
    auto xBlocksInfo = bpReader.AllStepsBlocksInfo(x_meta_info);
    auto x_it = xBlocksInfo.begin();
    
    for (; x_it != xBlocksInfo.end(); ++x_it) {
        const auto &var_vec1 = x_it->second;
        std:cout << var_vec1.size() << std::endl;  // as block number
    }

[Change72]

I also following openPMD Backend-Specific Controls

'Due to performance considerations, the ADIOS2 backend configures ADIOS2 not to compute any dataset statistics (Min/Max) by default. Statistics may be activated by setting the JSON parameter'

adios2.engine.parameters.StatsLevel = "1".

and following Warpx Full Diagnostics

<diag_name>.adios2_engine.parameters.* optional,

and I try

diag1.format = openpmd # must be openpmd
diag1.openpmd_backend = bp # bp
diag1.openpmd_encoding = g

diag1.adios2_engine.parameters.StatsLevel = "1"
or 
diag1.adios2_engine.parameters.StatsLevel = 1

However, in bpls -l output, it still all zero.

[ax3l]

Interesting... do you get a warning that the input diag1.adios2_engine.parameters.StatsLevel is unused or so?

@guj we might need to debug this.

[guj]

The parameter should be valid. Besides, currently the default in ADIOS is having stats level to be 1.
Do you know which version of ADIOS you use?

[Change72]

Hi, I figured this out by set diag1.adios2_engine.type = bp4

All related configurations are:

diag1.format = openpmd
diag1.openpmd_backend = bp 
diag1.openpmd_encoding = g
diag1.adios2_engine.type = bp4
diag1.adios2_engine.parameters.StatsLevel = 1

[ax3l]

Hi @Change72, I iwll try to answer question by question.

1. Step Encoding

Since I want to use ADIOS 2 as the backend, the only thing I changed is:

Looks good. Please note that groupBased encoding (g) is not recommended for BP4, but will work with our upcoming BP5 implementation:
https://openpmd-api.readthedocs.io/en/0.15.2/backends/adios2.html#bp4-file-engine

Until then, please use fileBased (f) and variableBased (v) encoding of steps.

2. Min/Max

After generation, I use bpls -l to check the min-max value, but it shows all 0.

yes, that is because we deactivated the min/max analysis in ADIOS2.
openPMD/openPMD-api#831
The data is still there, just these metadata values are not set.

My question is whether there will be some kind of overheads in collecting min/max if WarpX using Adios 2 backend, like 'ParticleExtrema'? If so, does such a cost is negligible or significant?

We need to remeasure it again
ornladios/ADIOS2#2880
and check cc @guj.

We have an environment variable
https://openpmd-api.readthedocs.io/en/0.15.2/backends/adios2.html#backend-specific-controls
that you can set if you like to have that analysis.
(or json option)

3. GPU runs & Block Size

Does this mean if I want to achieve near-optimal performance with 4 nodes and each of them has 2 GPUS, I need at least set it to 8 grids or say 8x grids, like 16 or 24?

Without load balancing, just use one block per GPU.

Otherwise with load balancing, 4-12 blocks per GPU are a good idea, if your problem is large enough to actually fill the GPU memory.

Details:

• https://warpx.readthedocs.io/en/latest/usage/workflows/domain_decomposition.html
• https://doi.org/10.1145/3468267.3470614

[ax3l]

3. Output formats

what is the relationship of AMReX and ADIOS2 in WarpX? As far as I know, AMReX output format is plotfile and ADIOS2 is bp files. It seems they are two different I/O formats. However, in the last page of ADIOS2 talk, it tells Codes such as the WarpX code, which uses AMReX can take advantage of ADIOS-BP4 for “fast” writing. So I am a little puzzled now.

Here is the overview:
https://warpx.readthedocs.io/en/latest/dataanalysis/formats.html

WarpX can generate output for "full diagnostics" (heavy data) with either:

• AMReX plotfiles (current default) or
• openPMD files.

openPMD supports backends HDF5 and ADIOS2.
ADIOS2 supports BP4 and BP5, and a few non-file transport methods, most notably for us SST: https://doi.org/10.1007/978-3-030-96498-6_6

We currently use ADIOS2 with BP4 and will change the default to BP5 soon.

We will in the near future also change the WarpX default from plotfiles to openPMD.