OpenPMD output capabilities

[MaxThevenet]

When running large simulations, writing the data into plotfiles is extremely fast, but reading the data, either particle data or grid data, can be time-consuming. One reason is that the data is spread in multiple boxes, to be able to account for mesh refinement. For many application, even when the simulation runs with mesh refinement, it would be useful to have a lower-resolution output using the OpenPMD standard, and if possible the hdf5 format, that would be easier and faster to read.

Furthermore, plotfiles can be massive as all grid quantities and all particles of all species are written to disk. Having the possibility to chose the data written to plotfiles would also be helpful.

At the end of a simulation, two folders would be created: plotfiles and diags (this name is arbitrary). Here are the details of what this issue proposes for each of these output:

plotfiles output

Currently, all grid data and all particle data are dumped, so that plotfiles contain grid quantities Ex, Ey, Ez, Bx, By, Bz, jx, jy, jz and particle quantities particle_Ex, particle_Ey, particle_Ez, particle_Bx, particle_By, particle_Bz, particle_momentum_x, particle_momentum_y, particle_momentum_z, particle_position_x, particle_position_y, particle_position_z, particle_cpu, particle_id, particle_xold, particle_yold, particle_zold, particle_uxold, particle_uyold, particle_uzold and particle_weight for all species. It would be useful if the user could specifies the species to dump and which particle and field quantities to dump, e.g. with

amr.plot_fields    = Ex, Ez, By, jz
amr.plot_species   = electron ion
amr.plot_particles = particle_position_z, particle_momentum_uz

Output compliant with the OpenPMD standard

The simulation directory would contain a second output directory, say diags, that contains a smaller amount of pre-processed OpenPMD-compliant output. Two types of output are considered below for a 3d simulation: sub-sampled arbitrary-dimension grid data and filtered particle data. The user would specify the name of each diagnostics with

warpx.diagnostics = dump3d line particle_dump
dump3d.type = grid_diagnostics
line.type = grid_diagnostics
particle_dump.type = particle_diagnostics

so that a user could define several grid_diagnostics objects, etc. Here is a more detailed description of each of these:

grid_diagnostics

The user specifies the fields and the output number of points as well as the boundaries of the data to dump, for instance

dump3d.fields = Ex, Ez, By
dump3d.n_cell = 128 64 64 # write 128 points in x and 64 in y and z.
dump3d.lo = -5.e-6 -5.e-5 -30.e-6
dump3d.hi =  5.e-6  5.e-5 120.e-6

for a 3d grid or

line.fields = Ex, Ez, By
line.n_cell = 1 1 64 # write 128 points in x and 64 in y and z.
line.lo = 0 0 -30.e-6
line.hi = 0 0 120.e-6

for a 1d line. The same syntax would also allow 2d slices.

Particle filters

The user specifies the species and can chose a simple filter, for instance

particle_dump.species = beam
particle_dump.filter = "uz > 10."

Note that the specific options to enter in an input file are still to be determined, as they should comply with the PICMI standard, so the choice proposed above is not permanent.

The user may want to dump small data with high frequency (for instance retrieving a line-out at every time), which will probably require buffering in order not to ruin the simulation performances.

Future improvements

• Allow grid_diagnostics to return data on a grid of arbitrary dimension that is not aligned with the simulation box. For instance, a user may want to return the Ex field on a line that goes through (0, 0, 0) with direction (1, 0, 1).
• Allow grid_diagnostics to return data on a set of points that are not aligned, for instance on a sphere or the arc of a circle.

[RemiLehe]

Most of the features mentioned in this issue are implemented. I added a new issue #428 for the particle filters.