GridSpacing: Per-Component UnitSI & UnitDimension #193

RemiLehe · 2018-04-23T15:48:47Z

Implements issue: Close #122

Description

This allows meshes to have axes with different dimensions (e.g. for phase space data where one axis is a position, and the second axis is a momentum.)

What is introduced, removed or renamed and why?

The attributes gridUnitDimension and gridUnitSI are now N times longer, where N is the dimension of the mesh.
In particular gridUnitDimension is a 1D array of length 7*N. This was preferred over a 2D array of dimension (N,7) since it seems that h5py does not support multidimensional arrays as attributes.

Affected Components

base: All mesh data

Writer Changes

How does this change affect data writers?
Data writers should now write the longer array for gridUnitSI and add gridUnitDimension.

Note: We could make this backward compatible by saying that, if the writer wrote the previous short-array format, then this applies to each axis... @ax3l Is this worth it or too confusing. -> too confusing, we have the updater for that :)

openPMD-api (upcoming): https://github.com/openPMD/openPMD-api/...

Reader Changes

How does this change affect data readers?
Data readers should now extract the dimension of each axis.

openPMD-validator: Create arrays that represent the dimension of each grid axis openPMD-validator#40

Data Converter

How does this affect already existing files with previous versions of the standard?
For existing attributes should be duplicated N times, in order to convert to the new format.

GridSpacing: Per-Component UnitSI & UnitDimension openPMD-updater#7

ax3l · 2018-04-24T13:06:14Z

Note: We could make this backward compatible by saying that, if the writer wrote the previous short-array format, then this applies to each axis...

Very good point. I think it's too special to allow this short-cut which just adds extra-checks in readers and writers. So let's not allow to just write it once as it was allowed before.

ax3l · 2018-04-24T13:07:24Z

STANDARD.md

@@ -95,7 +95,7 @@ Each file's *root* group (path `/`) must at least contain the attributes:
      present standard (e.g. fields on an unstructured mesh),
      this can be done be storing this data within a path that *is not*
      of the form given by `basePath` (e.g. `/extra_data`). In this
-      way, the openPMD parsing tools will not parse this additional data. 
+      way, the openPMD parsing tools will not parse this additional data.


while we are on it: it's not only the openPMD parsing tools but general openPMD data readers

ax3l · 2018-04-24T13:09:03Z

STANDARD.md

+    - examples:
+        - For a 2D spatial grid (`L=1`), store array
+        `(1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.)`
+        - For a 2D phase space (Unit "m" along the first axis


Maybe we better stay with L and M * L / T as defined in unitDimension?

ax3l · 2018-04-24T13:13:44Z

STANDARD.md

@@ -424,11 +424,29 @@ meshes):
    - example: `(0.0, 100.0, 0.0)` or `(0.5, 0.5, 0.5)`

  - `gridUnitSI`
-    - type: *(float64 / REAL8)*
+    - type: 1-dimensional array containing N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation


This is related to #125 & #129 (PR in #194): if we flatten the array we have to specify how.
I would propose to use the wording from #129 as such:
, dimensions shall be ordered from fastest to slowest varying index [of the described mesh].

update: we probably will go from slowest to fastest in #194

ax3l · 2018-04-24T13:14:20Z

STANDARD.md

+
+  - `gridUnitDimension`
+    - type: array of 7 N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation


we need to add the same comment with order as above for flattening

DavidSagan · 2018-11-07T17:19:46Z

STANDARD.md

+    - example: `(1.0e-9, 1.0e-9, 1.0e-6)`
+
+  - `gridUnitDimension`
+    - type: array of 7 N *(float64 / REAL8)*


My thought would be to not even mandate any precision (ie just say "real"). For example, why should the standard disallow the use of real4?

The unit dimension attribute is not a record but just describing the dimensionality ("unit"). It introduces no problem keeping it fixed and makes parsing very simple.

@RemiLehe pls replace array with 1-dimensional array for consistency.

Suggested change

- type: array of 7 N *(float64 / REAL8)*

- type: 1-dimensional array of 7 N *(float64 / REAL8)*

ax3l

Annotated the changed index order which is introduced in #194

ax3l · 2018-05-15T15:34:33Z

STANDARD.md

@@ -425,29 +425,30 @@ meshes):

  - `gridUnitSI`
    - type: 1-dimensional array containing N *(float64 / REAL8)*
-            elements, where N is the number of dimensions in the simulation
+            elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from fastest to slowest varying index of the described mesh.


will be inverted

ax3l · 2018-05-15T17:28:53Z

STANDARD.md

+
+  - `gridUnitDimension`
+    - type: array of 7 N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from fastest to slowest varying index of the described mesh.


slowest to fastest

ax3l · 2018-05-15T17:29:08Z

STANDARD.md

@@ -424,11 +424,30 @@ meshes):
    - example: `(0.0, 100.0, 0.0)` or `(0.5, 0.5, 0.5)`

  - `gridUnitSI`
-    - type: *(float64 / REAL8)*
+    - type: 1-dimensional array containing N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from fastest to slowest varying index of the described mesh.


slowest to fastest

ax3l

Changes look great!

Let us just add the PR to the converter before we merge.

PrometheusPi · 2018-05-24T08:54:47Z

@ax3l Does this require changing how we write hdf5 data in the phase space and radiation plugin in PIConGPU?

ax3l · 2018-05-24T10:03:35Z

@PrometheusPi there is a whole bunch of changes that are changing openPMD in a incompatible way in 2.0.0. This is one of them and will enable that we can describe phase space and radiation plugin in valid openPMD.

When we have all changes incorporated, a changelog will summarize all of them again. Also, we can then use openPMD-api instead of libSplash to write from all PIConGPU plugins.

We will then update all places of PIConGPU and then add openPMD to phase space and radiation plugin.

ax3l · 2018-11-07T17:19:37Z

STANDARD.md

+
+  - `gridUnitDimension`
+    - type: array of 7 N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.


Overlong line. Maybe put "Dimensions shall be..." on the next line (indented to same block).

I would try to make this more clear how the arrays are stored by writing:

"The order of the N 7-value arrays must be identical to the axes in axisLabels."

This makes it consistent with the other attributes above that have the same order and uses the same wording, also consistently with #194.

Suggested change

elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.

elements, where N is the number of dimensions in the simulation.

The order of the N 7-value arrays must be identical to the axes in `axisLabels`.

ax3l · 2018-11-07T17:42:11Z

STANDARD.md

@@ -430,11 +430,30 @@ In addition, the following attribute is *recommended* (see the section
 on Unit Systems and Dimensionality, further below):

  - `gridUnitSI`
-    - type: *(float64 / REAL8)*
+    - type: 1-dimensional array containing N *(float64 / REAL8)*
+            elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.


Suggested change

elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.

elements, where N is the number of dimensions in the simulation.

The order of the N values must be identical to the axes in `axisLabels`.

Apply changes as approved in last VC :)

RemiLehe mentioned this pull request Apr 23, 2018

Create arrays that represent the dimension of each grid axis openPMD/openPMD-validator#40

Merged

Store units/dimension of each axis

e77719d

RemiLehe force-pushed the per_axis_dimension branch from 95c41f5 to e77719d Compare April 23, 2018 15:53

RemiLehe assigned ax3l Apr 23, 2018

RemiLehe added the major change non-backwards compatible change label Apr 23, 2018

ax3l self-requested a review April 24, 2018 13:02

ax3l added this to Proposed in 2.0.0 via automation Apr 24, 2018

ax3l reviewed Apr 24, 2018

View reviewed changes

ax3l moved this from Proposed to Review in 2.0.0 Apr 24, 2018

Take into account review comments

d7d6f3f

ax3l requested changes May 15, 2018

View reviewed changes

RemiLehe added 2 commits May 15, 2018 21:11

Merge branch 'upcoming-2.0.0' into per_axis_dimension

c35a362

Switch convention from F order to C order

89e5c6b

ax3l approved these changes May 16, 2018

View reviewed changes

ax3l mentioned this pull request Nov 7, 2018

Put in **Definitions** section in the Standard #187

Open

ax3l reviewed Nov 7, 2018

View reviewed changes

ax3l mentioned this pull request Nov 7, 2018

GridSpacing: Per-Component UnitSI & UnitDimension openPMD/openPMD-updater#7

Merged

ax3l changed the title ~~Close #122 GridSpacing: Per-Component UnitSI & UnitDimension~~ GridSpacing: Per-Component UnitSI & UnitDimension Nov 8, 2018

RemiLehe mentioned this pull request Nov 8, 2018

Make "time", "dt", "timeOffset" and "timeUnitSI" optional #201

Open

7 tasks

ax3l force-pushed the per_axis_dimension branch from 554f7be to 7ea2a94 Compare November 13, 2018 09:09

gridSpacing/unitSI: as in axisLabels

4041dc0

Apply changes as approved in last VC :)

ax3l force-pushed the per_axis_dimension branch from 7ea2a94 to 4041dc0 Compare November 13, 2018 09:11

ax3l approved these changes Nov 13, 2018

View reviewed changes

ax3l merged commit 8223062 into openPMD:upcoming-2.0.0 Nov 13, 2018

2.0.0 automation moved this from Review to Implemented Nov 13, 2018

RemiLehe mentioned this pull request Dec 1, 2020

Add phase space diagnostics fbpic/fbpic#494

Open

franzpoeschel mentioned this pull request Dec 14, 2020

Use the openPMD API for the PhaseSpace plugin (formerly libsplash) ComputationalRadiationPhysics/picongpu#3468

Merged

6 tasks

This was referenced Oct 4, 2023

TransitionRadiation plugin openPMD output ComputationalRadiationPhysics/picongpu#4649

Open

WIP: Non-spatial meshes openPMD/openPMD-api#1534

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GridSpacing: Per-Component UnitSI & UnitDimension #193

GridSpacing: Per-Component UnitSI & UnitDimension #193

RemiLehe commented Apr 23, 2018 •

edited by ax3l

ax3l commented Apr 24, 2018

ax3l Apr 24, 2018

ax3l Apr 24, 2018

ax3l Apr 24, 2018 •

edited

ax3l May 15, 2018

ax3l Apr 24, 2018

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as resolved.

DavidSagan Nov 7, 2018

ax3l Nov 7, 2018 •

edited

ax3l Nov 7, 2018 •

edited

ax3l left a comment

ax3l May 15, 2018

ax3l May 15, 2018

ax3l May 15, 2018

ax3l left a comment

PrometheusPi commented May 24, 2018

ax3l commented May 24, 2018 •

edited

ax3l Nov 7, 2018

ax3l Nov 7, 2018 •

edited

ax3l Nov 7, 2018

	- type: array of 7 N (float64 / REAL8)
	- type: 1-dimensional array of 7 N (float64 / REAL8)

	elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.
	elements, where N is the number of dimensions in the simulation.
	The order of the N 7-value arrays must be identical to the axes in `axisLabels`.

	elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.
	elements, where N is the number of dimensions in the simulation.
	The order of the N values must be identical to the axes in `axisLabels`.

GridSpacing: Per-Component UnitSI & UnitDimension #193

GridSpacing: Per-Component UnitSI & UnitDimension #193

Conversation

RemiLehe commented Apr 23, 2018 • edited by ax3l

Description

Affected Components

Writer Changes

Reader Changes

Data Converter

ax3l commented Apr 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ax3l Apr 24, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as resolved.

Choose a reason for hiding this comment

ax3l Nov 7, 2018 • edited

Choose a reason for hiding this comment

ax3l Nov 7, 2018 • edited

Choose a reason for hiding this comment

ax3l left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ax3l left a comment

Choose a reason for hiding this comment

PrometheusPi commented May 24, 2018

ax3l commented May 24, 2018 • edited

Choose a reason for hiding this comment

ax3l Nov 7, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RemiLehe commented Apr 23, 2018 •

edited by ax3l

ax3l Apr 24, 2018 •

edited

ax3l Nov 7, 2018 •

edited

ax3l Nov 7, 2018 •

edited

ax3l commented May 24, 2018 •

edited

ax3l Nov 7, 2018 •

edited