Consider updating EML’s unitDictionary list

[mobb]

There are several issues logged which relate to the list of units built into EML. It would help constructors to have these cleaned up. The LTER created a unit resource so that constructors can reuse custom units created by other LTER sites, and see examples. It is a RDB (interface at http://unit.lternet.edu) containing both the built-in units and units contributed by many LTER sites. LTER also created guidelines for units best practice (for both their dictionary and EML), but these have not yet been routinely applied to the dictionary.

EDI needs a unit list for its own EML construction, and the LTER unit dictionary is a candidate. A “clean” list of units from the LTER-dictionary may be a good candidate for EML as well. So the following tasks benefit both groups.

Steps:

1. Migrate the LTER guidelines to the EDI website (side task, not directly related to EML)
2. finish the vetting of the units in the LTER dictionary
3. Export as stmml, ie, for eml-unitDictionary.xml
4. Examine for use in EML 2-2
5. Create a proposal for EML-2.2

Related to issues: #139, #140, #115, #284, #285, #286, #287

[mobb]

Other things to consider:

1. EML 2.2 will be backward compatible, so all the units currently in the list must be included, whether or not they match the guidelines.
2. One of the problems with the 2.1 list is that spellings of unitNames are inconsistent. The “unitName” is simply a string id which points to the definition, so from one standpoint, the spelling doesn’t matter. However, since it’s the only required component, it has become de facto, a meaningful version of the unit. Some of the proposed, vetted additions will simply be alternate spellings of 2.1 units, which is likely to cause confusion (if they show up together).
3. The LTER-dictionary has an attribute deprecatedInFavorOf, which (if adopted by EML2.2) could be used to shorten the list (by demoting non-preferred spellings, e.g., in an EML editor), while still allowing them in docs.
4. STMML is no longer widely used. Even the current version (stmml-1.1) was created by eml-dev for EML-2.1 (with permission from 1.0’s authors).

[mbjones]

@mobb agreed on all counts in your last two comments. I particularly like the addition of a deprecatedInFavorOf attribute, and I don't think its a problem to add it to STMML because we are the defacto maintainers of our fork.

[csjx]

Yes, while STMML is not widely used or maintained, we widely use it and maintain the fork as Matt said. I'm in favor of considering a replacement, but only if it is a superset of the functionality and expressivity of STMML, and does have wide use and maintenance. Replacing STMML will also involve a bunch of client software refactoring so that's something to consider. Did you have a replacement in mind @mobb, or were just noting how STMML kind of dead-ended from a maintenance/improvement perspective?

[mobb]

@csjx - nope - no replacement in mind. Just pointing out that to the best of my knowledge, we are the only ones using it.

[csjx]

For reference, the NCEAS data team has been investigating the udunits library from Unidata. From a quick scan of the XML instance documents, I don't see a formal schema, but it looks like some extensive work has been put into it and may inform our unit dictionary: https://www.unidata.ucar.edu/software/udunits/

[cboettig]

👍 for Unidata units. (The library also has nice R bindings). It also looks like it would be straight-forward to generate STMML unitList definitions from the the Unidata udunits2 XML library

[srearl]

As information managers increasingly use R and tools like ropensci::EML to generate metadata, an approach/framework that would allow for a tighter integration between R and services/packages that would aid documenting units would be very welcome.

[mobb]

Re udunits2:
I've played around with this a bit, and the database is pretty comprehensive, and conversions are pretty straightforward. It looks like the grammar can be mapped to stmml, eg, kilogrampermetersquared == kilogram per meter^2
Common notations (kg/m^3) are also supported, I presume as synonyms.

The mapping between these are probably best placed in the EML-unit dictionary; to do so in udunits2 would be a major fork.

I am not quite clear on their use of unit system. I think it maps to stmml unit type -- in that it represents a unique group of base units that a defined unit belongs to, so that conversions are not allowed between units in different systems. So the STMML unit type amountOfSubstanceConcentration is equivalent to the udunit system that is a quotient of [mole, volume]. Both volume quantities (quart, liter) and length^3 will work. I just can’t quite tell where these systems are defined. But if we simply include the mapping at the unit level, (per the above), udunit2 will take care of it.

I think we are better off not generating a list of STMML units from udunits, though. For a couple of reasons:

• We would still probably want to generate a unitType, and it’s not clear how
• Units are combinations of prefix+base, so there would be a whole lot lot of almost-impossible units generated, eg, petagram per centimeter^3
• Some of the older, arcane EML units are not in udunits2 (indianYards!) so manual work is still in order to get them in

That is debatable, of course. But unless someone has a good argument (and wants to do the coding), I will continue with the plan to vet the units in the LTER DB, and export the STMML from that. There may be other units we want to add to EML’s list, but the LTER DB is already backward compatible (with EML 2.1) and is a good representation of what is commonly used now.

Regarding a nascent stmml-1.2, there are now two candidates for optional attributes for a unit:

• deprecatedInFavorOf
• synonym_udunit2

[maier-m]

Udunits understands symbolic units. So synonym_udunit2 would just be the EML abbreviation I believe.

[mobb]

that occurred to me too - that abbreviation could hold this (be the synonym). but we have to watch out for character sets. using a dedicated attribute might make it more explicit.

[maier-m]

The character encoding for udunits2 is "US-ASCII". If there was a problem with characters, the dedicated attribute would then likely have to be the symbolic form of the unit spelled out with care to have numeric exponents... e.g. "meters3 per second".