No sign flip of equilibrium.time_slice[].boundary.psi upon conversion from DDv3 to DDv4

[Rick-Fusion]

No flip of sign is observed of the equilibrium.time_slice[].boundary.psi field when converting the IDS from DDv3.39.0 and DDv3.42.0 to DDv4.0.0, while it should.

[maarten-ic]

Good catch Rick! The issue is that DD 3.x did not indicate a cocos_label_transform for this quantity, therefore it's not covered by the conversion rules.

There seem to be some more quantities in equilibrium for which this is a problem:

• time_slice/boundary/psi
• time_slice/boundary_separatrix/psi / time_slice/boundary_secondary_separatrix/psi, but these aren't covered by the DD conversion right now (see conversion of boundary_separatrix and boundary_secondary_separatrix data in dd 3.42 to contour_tree data in dd 4.0 #60 )
• time_slice/global_quantities/q_min/psi
• time_slice/ggd/psi

@olivhoenen / @imbeauf Since retroactively updating the DD is not feasible, we should probably add another DD3to4 rule to IMAS-Python, which hardcodes transformation rules for these quantities?

Could there be missed cases in other IDSs as well?

[imbeauf]

@olivhoenen , please comment on the solution proposed by Maarten

[SimonPinches]

Since I think we can’t rule out having made or making other mistakes in handling the conversions between DD versions in the future, I wonder if it wouldn't be better to correct the conversion metadata as part of making new releases of the data_dictionaries package so that IMAS-Python remains unchanged.

[olivhoenen]

I get both points from @maarten-ic and the alternate solution proposed by @SimonPinches. While finding a way to silently patch the faulty DD without updating the code, I find it risky to start having an XML reference that may be different between IMAS-Data-Dictionary and IMAS-Data-Dictionaries. This change is going to impact only the DD3->DD4 conversion case, and IMAS-Python already have a dedicate function for that, so for me it's ok to extend it.

[SimonPinches]

This specific change is only going to impact the DD3 to DD4 conversion in IMAS-Python but I don't think we can rule out finding more errors in the future between different major or minor versions of the DD, or even having more tools than just IMAS-Python that do conversions.

For me, IMAS-Data-Dictionaries is the source of metadata for converting between DD versions. (That's why it provides access to multiple versions in once package.) The conversion metadata in IMAS-Data-Dictionary is useful for knowing how to transform from external non-IMAS sources. We can't correct mistakes in this and can only make new releases. Users mapping external data to a DD version with erroneous metadata will have to correct the mistake themselves, e.g. in their JSON mapping file.

Having said all this, I get the point of @olivhoenen about having XML references that differ between IMAS-Data-Dictionary and IMAS-Data-Dictionaries, however, if we restrict to only correcting transformation metadata, then I think this might be okay.

[olivhoenen]

I'm not sure to understand the example about mapping external data. This is done for a given version of the DD, then the hope is that there are other tools doing the conversions (with necessary patch when issues are found in the DD metadata) from version N to N+x.

We can give a try at patching IMAS-Data-Dictionaries, but this is not without risk (a bug can be introduced while applying a patch to 20+ different XML files, each of which is >10MB) and have the potential to introduce a "but it worked yesterday" effect.

If we want to fix the DD, I would prefer to push a release DD 3.42 patch, and have the DD3to4 always done by first convert to latest v3 before applying the v4 rules, which should gives us the best chances at migrating through a major version bump (but this would need to be check with @maarten-ic for its implication on ids_convert from IMAS-Python).

[maarten-ic]

We can give a try at patching IMAS-Data-Dictionaries, but this is not without risk (a bug can be introduced while applying a patch to 20+ different XML files, each of which is >10MB) and have the potential to introduce a "but it worked yesterday" effect.

I also prefer to not edit the IMAS-Data-Dictionaries XML files.

If we want to fix the DD, I would prefer to push a release DD 3.42 patch, and have the DD3to4 always done by first convert to latest v3 before applying the v4 rules

This would be possible without too much impact on the underlying code. However, it will make the conversion consume more memory (for the intermediate IDS object) and take roughly twice as long.
Though I wonder how useful it will be: the only conversion rule that can be fixed with this approach is the psi sign flip rule. The other conversion rules are specific to certain IDSs/paths, or depend on the metadata of the newer DD release (e.g. NBC rules).

[olivhoenen]

Ok, then let's add the extra dedicated rule in DD3to4 function as you initially proposed @maarten-ic