WIGOS metadata integration: referencing a set of stations in a WCMP2 record based on a network of stations

[tomkralidis]

@wmo-im/tt-wismd @wmo-im/tt-wigosmd @joergklausen @steingod @thineshsornalingam

Consider a WCMP2 record describing, say, surface weather observations from a network of stations/facilities. In WCMP2, we are able to enumerate 1..n links, i.e:

(NOTE: relation types are examples/for brainstorming)

{
    "rel": "http://www.wmo.int/def/rel/wmdr/1.0/ObservingFacility",
    "type": "application/xml",
    "title": "Station report for Thessaloniki (Greece)",
    "href": "https://oscar.wmo.int/surface/rest/api/wmd/download/0-20008-0-THE"
}

In the above, a user can discover a dataset collection and determine from the link relation (rel) the association station(s) providing the dataset. Now imagine a observing network with 800 stations.

WCMP2 requires an approach whereby a data provider can provide a facility set as a single link in order to scale.

{
    "rel": "http://www.wmo.int/def/rel/wmdr/1.0/FacilitySet",
    "type": "application/xml",
    "title": "Station report for facility set 123",
    "href": "TBD"
}

Questions:

1. what do folks think about this approach? This would require data providers to define such sets in OSCAR/Surface
2. can the OSCAR/Surface API provide a link a facility set?

Aside: while the above examples use the OSCAR/Surface API, in theory the value of the href is opaque here, so long as a WMDR facility set is provided in the response.

Looking forward to this important discussion.

[efucile]

There is already a requirement to identify the stations as being part of GBON for a country. Not sure if this will be accessible through the API in a convenient way. @kurt-hectic can you give some info on this?

[joergklausen]

The OSCAR/Surface API is a non-standard API, so in principle, parameterized endpoints can be defined ad libitum. The present API supports direct access

• to a single station, viz.

  https://oscar.wmo.int/surface//rest/api/wmd/download/<WIGOS_ID>

• to a whole country, viz.

  https://oscar.wmo.int/surface/rest/api/search/station/?territoryName=<notation from http://codes.wmo.int/wmdr/TerritoryName>

• to all stations affiliated with a program/network, viz.

  https://oscar.wmo.int/surface/rest/api/search/station/?programAffiliation=<notation from http://codes.wmo.int/wmdr/ProgramAffiliation>

• combinations are possible, e.g.

  https://oscar.wmo.int/surface/rest/api/search/station/?programAffiliation=GAW&territoryName=CHE

See User Manual for reference.

I believe, the endpoint to expose station sets is going to be available later in 2022.

The question could be if an OGC standard API like STA should be supported in future.

[tomkralidis]

Thanks @joergklausen. It would be valuable to have the facility set use case realized in the API. As an aside, we would be interested in discussions around OSCAR API future plans, and would suggest that the OGC API suite of standards is considered.

[tomkralidis]

To summarize an option:

• Station metadata SHALL be managed in OSCAR/Surface
• WIS2 Discovery metadata SHALL be managed by the data producer and be published to the WIS2 Global Discovery Catalogue (GDC)
• When WIS2 metadata describes a dataset which is comprised of observing stations, WIS2 metadata SHOULD encode via the following options:
  1. Provide a link to each station directly
  2. Provide a single link to a facility set (group) of 1..n stations

Whether to choose option i or ii above depends on the number of stations that a given dataset is based on. For example, for ~800 surface stations in Canada, we would use option 2.

In both cases:

• station metadata is not required to be provided beyond a reference in WIS2 metadata
• the OSCAR/Surface API needs to provide endpoints for both options, and needs to be ready for traffic accordingly from WIS2 GDC discovery

[tomkralidis]

TT-WISMD 2023-05-05: ACTION @wmo-im/tt-wismd to review and provide comments.

[josusky]

@tomkralidis your summary matches what we have discussed earlier and there is little I can add to it. Clearly, option 2 will be a great help and the only open question is if the existing OSCAR API station filtering capability is able to cover all cases. And if not, whether the right way is to ask for its extension or whether it is then the task of the publisher to provide an API (possibly a facade over OSCAR) that will return the right list of stations.

[steingod]

@tomkralidis I have nothing to add to what you commented on. Although I doubt that station metadata are managed in OSCAR/Surface I would assume many provide a copy to OSCAR where it is published and made available. But that is not the major issue here. As long as both options for linking WIS and WIGOS are possible it is positive although I find the option of direct relation between station dataset (WIS) and station metadata the most useful for services targeting decision support through integration in decision support systems at various levels. It will be interesting to see if WMO information services will be embraced in this context.

[tomkralidis]

Thanks for the feedback. At this point we'll add an example WCMP2 record to this effect that will become a curated example as part of an Annex in WCMP2.

[tomkralidis]

For station sets, given implementation in OSCAR/Surface is pending next generation discussions, perhaps an additional option can be to point to a list of stations in any format:

{
    "rel": "related",
    "type": "text/csv",
    "title": "Dataset station list",
    "href": "https://dd.weather.gc.ca/observations/doc/swob-xml_station_list.csv"
}

Or:

{
    "rel": "related",
    "type": "application/json",
    "title": "Dataset station list",
    "href": "https://example.org/oapi/collections/stations/items?f=json"
}

Note: we could (in WCMP2) perscribe a dedicated link relation (e.g. http://wis.wmo.int/metadata/stations) for tighter integration with WIS2 tooling, at the cost of broader interoperability.

[joergklausen]

Even the current implementation of station clusters / station ets / FacilitySet in OSCAR/Surface exposes a URL, e.g., https://oscar.wmo.int/surface/#/search/stationClusterReport/5. Since there is no WMO/WIGOS identifier for this, the identifier used (here: 5) will probably not survive the current OSCAR/Surface implementation, though. The requirement has been noted for OSCAR NextGen.

[amilan17]

https://github.com/wmo-im/tt-wismd/wiki/Meeting-2023-08-14 notes:

Tom: should we include a section in the Guide for this or should this be a part of best practices? 

• link relations can be defined as needed and we can add one called "station set" or ... 
• Jan: this is only relevant for the station or upper air data observations;

[tomkralidis]

TT-WISMD 2023-08-14

• example records will be useful
• may have a specific link relation for station or station list (facilityset)
• keep agnostic to representation
• ACTION: @tomkralidis to define link relation for stations

[tomkralidis]

Initial link relations put forth in wmo-im/wcmp2-codelists#4

We will need to discuss whether this is the best approach (would this be better managed in WMO Common Code Tables?), as well as the publication to codes.wmo.int (which would then be the canonical reference via infrastructure).

cc @david-i-berry

[tomkralidis]

Notes following discussion with ET-WTR (cc Cristina Prates @thineshsornalingam):

We want to reduce duplication between WIS2 and OSCAR. Notes:

• observed properties in WMDR can be used as WCMP2 themes/concepts to help w/ discovery
• WCMP2 by nature supports this already
• The WIS2 Global Discovery Catalogue (GDC) minimally touches the following WCMP2 metadata elements on ingest:
  • global services links
• The GDC could additionally augment a WCMP2 record as follows:
  • determine whether the WCMP2 record is based on observations (looking for Facility or FacilitySet links in same), in which case:
    • foreach WSI
      • query OSCAR/Surface API for the WSI report
      • add observed property names as themes/concepts

The result would be a requirement or recommendation on GDCs to augment WCMP2 on ingest per the above.

Considerations:

• could this logic be applied to other domains (say GRIB2 codenames for NWP, etc.)?
• this results in a workflow update in the architecture of WIS2 for catalogue ingest
• we also need to discuss OSCAR/Surface API (both the interface specification as well as availability)

[tomkralidis]

Updated examples based on new link relations:

Single facility:

{
    "rel": " http://def.wmo.int/def/rel/wmdr/-/EnvironmentalMonitoringFacility",
    "type": "application/xml",
    "title": "Station report for Thessaloniki (Greece)",
    "href": "https://oscar.wmo.int/surface/rest/api/wmd/download/0-20008-0-THE"
}

Facility Set

{
    "rel": "http://def.wmo.int/def/rel/wmdr/-/FacilitySet",
    "type": "application/xml",
    "title": "Station report for facility set 123",
    "href": " https://oscar.wmo.int/surface/rest/api/facilityset/123"
}

[tomkralidis]

TT-WISMD 2023-09-13:

• some information may be in both systems
  • observed properties
  • contacts
• how do we ensure consistency?
• need to clarify RTH workflows
  • WIS2 eliminates the RTH concept
• needs awareness/training issues
• data metadata
• not always consistent
• broader ET-Data/TT-TDCF issue
• bi-directional
• discovering WIS2 metadata from WIGOS or OSCAR/Surface
• can we do nested FacilitySets?
  • can be put forth as 1..n links, for example
• need to integrate satellite observations
  • add examples
• ET-WDR: put forth requirements to SC-ON, possible discussion with SC-IMT

[tomkralidis]

TT-WISMD 2023-10-06:

• @tomkralidis to add informative note to WCMP2

[amilan17]

WMDR codes currently only has codes for Facility Type, should there be a request to expand this code list or create a new codes list in WMDR to support this type of linkage? 

https://codes.wmo.int/wmdr/FacilityType

[amilan17]

We will need to discuss whether this is the best approach (would this be better managed in WMO Common Code Tables?), as well as the publication to codes.wmo.int (which would then be the canonical reference via infrastructure).

Alternatively, I like the idea of using common code tables, but it's currently linked to WMO-No. 306, vI.2 https://codes.wmo.int/common

[tomkralidis]

Current list in https://github.com/wmo-im/wcmp2-codelists/blob/main/codelists/link-type.csv

Options:

1. define entities in each specification, and publish to codes.wmo.int
2. define in a "common/shared" vocabulary
   a. WMO 306, or something else)
   b. another vocabulary (WIS2 concepts?)
3. other options?

Note that WMO 306 (https://github.com/wmo-im/cct) is for data specific code tables, whereas we are trying to describe higher level things.

[amilan17]

@joergklausen may have ideas on how to manage shared vocabularies across WIS and WIGOS metadata....

[joergklausen]

WIGOS was born to cover common topics of interest, specifically for observations. So, I am fully in support of sharing common vocabularies. In fact, we should clarify what we mean by common ... between whom? The respective metadata models shold refer to the correct name space, which for WIGOS doesn't always have to be wmdr. Of course, we need clean governance structures.

[amilan17]

https://github.com/wmo-im/tt-wismd/wiki/Meeting-2023-10-23 notes:

• Tom talked about an example from NASA that used multipoint geometries for a station set;
• Tom will add section to the WCMP2 spec

[tomkralidis]

I started drafting a section in WCMP2 (clause "stations/observations"), and I'm conflicted. If we add a specific section for "stations/observations", does this introduce precedent to add more sections for NWP, Hydrology, Cryosphere, etc.?

Options:

1. in WCMP2 proper, introduce a section foreach discipline, starting with "stations/observations" (over time)
2. discipline specific information goes into a "best practice/guidance" document (which we will start after the INFCOM-3 deadline). For INFCOM-3, build out example documents (for "stations/observations" and other disciplines). If specific elements/properties are added to WCMP2 proper, then they are obviously added accordingly

Thoughts? cc @antje-s @jsieland @josusky @Amienshxq @solson-nws @gaubert @amilan17 @masato-f29 @McDonald-Ian

[tomkralidis]

TT-WISMD 2023-11-16: add similar example from OceanOPS, for example.