Common, Measures and other cross-package features Readiness

[blchoy]

Referencing #245.

We (Jan and Choy) propose the following implementation of extension for code list and enumeration classes:

Code list class:

[Post proposal note: Multiplicity of extension has changed from [0..1] to [0..*] to align with existing extensions in accordance to comments by @moryakovdv and @mgoberfield in the following posts]

<complexType name="AerodromePresentWeatherType">
	<annotation>
		<documentation>...</documentation>
		<appinfo>
			<vocabulary>http://codes.wmo.int 49-2/AerodromePresentOrForecastWeather</vocabulary>
			<extensibility>none</extensibility>
		</appinfo>
	</annotation>
	<complexContent>
		<extension base="gml:ReferenceType">
			<sequence>
				<element name="extension" type="gml:AssociationRoleType" minOccurs="0" maxOccurs="unbounded"/>
			</sequence>
		</extension>
        </complexContent>
</complexType>

The following is a possible instance fragment:

<iwxxm:presentWeather nilReason="http://codes.wmo.int/common/nil/inapplicable">
	<iwxxm:extension xlink:href="http://your.own.codes.registry/your/own/code/list/entry"/>
</iwxxm:presentWeather>

Enumeration class:

<simpleType name="TropicalCycloneMovementType">
	<annotation>
		<documentation>...</documentation>
	</annotation>
	<union>
		<simpleType>
			<restriction base="string">
				<enumeration value="MOVING">
					<annotation>
						<documentation>...</documentation>
					</annotation>
				</enumeration>
				<enumeration value="STATIONARY">
					<annotation>
						<documentation>...</documentation>
					</annotation>
				</enumeration>
			</restriction>
		</simpleType>
		<simpleType>
			<restriction base="string">
				<pattern value="extension:.+"/>
			</restriction>
		</simpleType>
	</union>
</simpleType>

and the instance fragement looks like:

<iwxxm:movement>extension:ABC</iwxxm:movement>

I would like to propose added extensions as above to all IWXXM code lists and enumerations. These changes are both forward and backward compatible. If we have a concurrence we shall be able to include this in FT2021-2. Your views please?

[mgoberfield]

Are you proposing these changes for all codelists and enumerations?

[blchoy]

Are you proposing these changes for all codelists and enumerations?

Yes. I have added this intention back to my previous post. In case you have any concern just voice out. You are our expert in extensions.

[moryakovdv]

Hello, may be I've missed something, but do we allow only one extension entry in the block?
<element name="extension" type="gml:AssociationRoleType" minOccurs="0" maxOccurs="1"/>

[blchoy]

Hello, may be I've missed something, but do we allow only one extension entry in the block?
<element name="extension" type="gml:AssociationRoleType" minOccurs="0" maxOccurs="1"/>

It is by design to have only one extension block per element. It applies not only to the new code list extension but existing extensions too. Our intention is not to overwhelm the original element by extensions:

<iwxxm:presentWeather nilReason="http://codes.wmo.int/common/nil/inapplicable">
	<iwxxm:extension>...</iwxxm:extension>
	<iwxxm:extension>...</iwxxm:extension>
	<iwxxm:extension>...</iwxxm:extension>
	<iwxxm:extension>...</iwxxm:extension>
        ...
</iwxxm:presentWeather>

As the extension has a type "Any" (implemented in XML with "gml:AssociationRoleType"), you can use existing or create your own schema to put more complicated instance fragment there.

[mgoberfield]

Actually, @blchoy and @moryakovdv, all of the complex types listed in the extension tutorial table allow for an unlimited number of extension elements to appear within them. However, I do agree with Choy that we should not allow IWXXM documents to be overwhelmed with numerous extension blocks and that is why there is a 5K character limit on extension block content to prevent such "abuse." Occasionally there are instances when US METARs have to have two extension blocks within a single complex type, so I am grateful that the current schemas allow this.

[mgoberfield]

I should add that I agree with @blchoy that a single instance should suffice for both code list and enumeration extensions as you've proposed above. It's hard for me to see why anyone would need multiple instances in these cases.

Also, may I kindly and respectfully suggest a different regular expression for extending enumerations? Perhaps something like
'[A-Z0-9_]{,30}' or is that too restrictive on character code points? Do we allow Cyrillic, Arabic, and Chinese characters to appear in IWXXM messages? If so, then '.+' is probably better.

[blchoy]

all of the complex types listed in the extension tutorial table allow for an unlimited number of extension elements to appear within them

@mgoberfield was right, the existing schema allows unlimited number of extensions! I cannot recall why I have "only a single extension is allowed" in my memory. The form of extension was determined by Aaron and may be what I have in my mind is my believe which had not been accepted? :) In any case we have a design decision to make:

1. Code list extension to follow existing multiplicity of extension of other classes, viz [0..*]. This is the natural choice.
2. Make multiplicity of all extensions [0..1]. This is in accordance with our latest design concept. However, this will also risk breaking somebody's extension plan.

Let's confirm this on Monday.

[blchoy]

Also, may I kindly and respectfully suggest a different regular expression for extending enumerations? Perhaps something like '[A-Z0-9_]{,30}' or is that too restrictive on character code points? Do we allow Cyrillic, Arabic, and Chinese characters to appear in IWXXM messages? If so, then '.+' is probably better.

I do not have a strong view on this, but I think what @mgoberfield proposed makes sense. In contrast to other extensions which only let people to introduce their own representation, the extension for enumeration allows one to add entries to an existing list of IWXXM selections. So it is natural for the added entries to follow IWXXM format, which is what Mark's regular expression is trying to constrain. In fact, if one would like to have enumeration entries in their own language, they may as well do this in an extension of other classes. Now we have another design decision to make on Monday:

1. To further limit enumeration extension entries to be in a format compatible with other IWXXM enumerations.
2. To have no limit to the added enumeration entries.

[mgoberfield]

While an unlimited number of extension elements is allowed by the schemas, in reality, the Schematron rule, Common.Report-2, limits its use significantly in IWXXM documents as it was designed to do. The unbounded number of extension elements has been in place since IWXXM v2.0. If option 2 is chosen, that will require a breaking change to the US schemas, i.e., a new major release which I want to avoid while we are introducing IWXXM to data consumers and the FAA.

[blchoy]

While an unlimited number of extension elements is allowed by the schemas, in reality, the Schematron rule, Common.Report-2, limits its use significantly in IWXXM documents as it was designed to do. The unbounded number of extension elements has been in place since IWXXM v2.0. If option 2 is chosen, that will require a breaking change to the US schemas, i.e., a new major release which I want to avoid while we are introducing IWXXM to data consumers and the FAA.

Do you think we also need to revise the the total number of characters across all extension blocks? As we are adding new extensions it is perhaps time to also raise this number. Is 10000 (2 times existing) good enough or you think more is better.

[mgoberfield]

I've been running TAC->IWXXM software on US METAR/SPECI observations in near real-time for the past year or so. Only once did the validation tool flagged a METAR message for exceeding the 5K limit. Let me check with the NCAR folks, they do the AIRMETs and SIGMETs TAC->IWXXM work. Unfortunately, we won't get a response from them before Monday's meeting.

[blchoy]

Only once did the validation tool flagged a METAR message for exceeding the 5K limit.

Seems to me this is yet to become an issue. Let's see if we can get a consensus in today's meeting first; the exact number can be finalized before I package RC1 for distribution.

[jkorosi]

I think that the limit on the number of extensions should be related only to code lists and enumeration types, but not to the already presented extensions.
@mgoberfield: I believe that solves the US IWXXM issue. Am I right?

I also like the @mgoberfield restriction '[A-Z0-9_]{,30}', maybe the length should be bigger, e.g.
<iwxxm:phenomenon xlink:href="http://codes.wmo.int/49-2/SigWxPhenomena/VA"/>
contains 43 characters. I don't know what should be the right length, maybe the length should not be restricted.

[blchoy]

I also like the @mgoberfield restriction '[A-Z0-9_]{,30}', maybe the length should be bigger

The template will only be used in enumeration. 30 characters is quite long already:
<iwxxm:movement>extension:012345678901234567890123456789</iwxxm:movement>

[mgoberfield]

I am not aware of another prolific user of extension elements comparable to the US. The situations which are likely to exceed the 5K limit are those which are of great significance and therefore the information should not be rejected. The one case I mentioned? It was a lengthy and complex METAR that occurred during a severe ice storm. For that reason, I am in favor of increasing the limit, but only a bit more, say, 6000 characters. I think NCAR will agree also.

[amilan17]

not ready. still under discussion