Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" #32

ywang-bom · 2014-11-18T02:11:58Z

NOTE: This bug concerns metadata records NOT describing data for global exchange, i.e. the data do NOT have a corresponding WMO Data License Code (e.g. WMOEssential).

OpenWIS refuses to create a metadata record because the system derives an empty string as the "GTS Category" from the given XML file.
The system reports error "Unable to extract OpenWIS information" shown as follows:

Steps to reproduce

Login to the user portal with an account that has editor right.
Navigate to "My Account" -> "Metadata Service" -> "Insert Metadata"
Choose the given test metadata for upload ( the content of the test metadata can found in the first comment to this ticket, simply copy and paste to create the test file)
Select "XSD only" from the "Validate" drop-down menu
Click "Insert"
The error report should be generated as shown above.

Analysis
From our investigation, OpenWIS has a set of logic to derive the value of "GTS Category" for a given XML file. The relevant method is extractGTSCategoryGTSPriorityAndDataPolicy in ProductMetadataISO19139Extractor.java. (The "GTS Category" corresponds to the "gts_category" column of "openwis_product_metadata" table in database)

The logic to derive "GTS Category" from a given file is list as follows:

Search for the existence of following two elements
* //gmd:identificationInfo//gmd:resourceConstraints//gmd:useLimitation
* //gmd:identificationInfo//gmd:resourceConstraints//gmd:otherConstraints
If none of the above two XML elements are found, set the "GTS Category" to Unknown and return.
If elements are found, loop through all elements and try to find match for WMO Data License Code (e.g. WMOEssential).
If WMO Data License Code is found, use it as the "GTS Category".
If WMO Data License Code is NOT found (source of the error), set "GTS Category" to the content of the last element in the loop.

Relevant config file is openwis-metadataportal.properties, which instructs OpenWIS to use the above two elements to derive "GTS Category".

The error is generated when:

The given metadata is NOT for global exchanged data (hence no WMO Data License code can be found)
Both elements,gmd:useLimitation and gmd:otherConstraints, can be found in the file.
gmd:useLimitation is an empty element (i.e. <gmd:useLimitation gco:nilReason="missing"> <gco:CharacterString /> </gmd:useLimitation>)
gmd:useLimitation is the last element in the loop.

The method extractGTSCategoryGTSPriorityAndDataPolicy will then set "GTS Category" to be an empty string, which is exactly the content of gmd:useLimitation. This in turn triggers the reported error generated by ensureProductMetadataValid in MetadataAligner.java.

Possible Fix
A possible fix is to check the "GTS Category" value at the end of the function when the given XML is NOT for global exchanged data. If the value is an empty string, simply set it to "Unknown" similar to step 2 in above logic.

The text was updated successfully, but these errors were encountered:

ywang-bom · 2014-12-02T04:28:08Z

Test Metadata for reproducing the bug

<?xml version="1.0" encoding="UTF-8"?>
<gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd"
                 xmlns:gco="http://www.isotc211.org/2005/gco"
                 xmlns:gml="http://www.opengis.net/gml/3.2"
                 xmlns:gmx="http://www.isotc211.org/2005/gmx"
                 xmlns:gts="http://www.isotc211.org/2005/gts"
                 xmlns:mdbase="http://www.bom.gov.au/wis/mdbase"
                 xmlns:geonet="http://www.fao.org/geonetwork"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="http://www.isotc211.org/2005/gmd http://www.ngdc.noaa.gov/metadata/published/xsd/schema/gmd/gmd.xsd http://www.isotc211.org/2005/gmx http://www.ngdc.noaa.gov/metadata/published/xsd/schema/gmx/gmx.xsd">
  <gmd:fileIdentifier xmlns:gml="http://www.opengis.net/gml">
      <gco:CharacterString>urn:x-wmo:md:au.gov.bom::ticket-32-test-1</gco:CharacterString>
  </gmd:fileIdentifier>
  <gmd:hierarchyLevel>
      <gmd:MD_ScopeCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#MD_ScopeCode"
                        codeListValue="series"/>
  </gmd:hierarchyLevel>
  <gmd:contact>
      <gmd:CI_ResponsibleParty>
         <gmd:organisationName>
            <gco:CharacterString>BoM</gco:CharacterString>
         </gmd:organisationName>
         <gmd:role>
            <gmd:CI_RoleCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#CI_RoleCode"
                             codeListValue="pointOfContact"/>
         </gmd:role>
      </gmd:CI_ResponsibleParty>
  </gmd:contact>
  <gmd:dateStamp>
      <gco:DateTime xmlns:gml="http://www.opengis.net/gml">2014-12-02T00:00:00</gco:DateTime>
  </gmd:dateStamp>
  <gmd:metadataStandardName>
      <gco:CharacterString>WMO Core Metadata Profile of ISO 19115 (WMO Core), 2003/Cor.1:2006 (ISO 19115), 2007 (ISO/TS 19139)</gco:CharacterString>
  </gmd:metadataStandardName>
  <gmd:metadataStandardVersion>
      <gco:CharacterString>1.3</gco:CharacterString>
  </gmd:metadataStandardVersion>
  <gmd:identificationInfo>
      <gmd:MD_DataIdentification>
         <gmd:citation>
            <gmd:CI_Citation>
               <gmd:title>
                  <gco:CharacterString>Ticket 32: test metadata 1</gco:CharacterString>
               </gmd:title>
               <gmd:date>
                  <gmd:CI_Date>
                     <gmd:date>
                        <gco:Date>1970-01-01</gco:Date>
                     </gmd:date>
                     <gmd:dateType>
                        <gmd:CI_DateTypeCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#CI_DateTypeCode"
                                             codeListValue="publication"/>
                     </gmd:dateType>
                  </gmd:CI_Date>
               </gmd:date>
            </gmd:CI_Citation>
         </gmd:citation>
         <gmd:abstract>
            <gco:CharacterString>abstract</gco:CharacterString>
         </gmd:abstract>
         <gmd:descriptiveKeywords>
            <gmd:MD_Keywords id="WMOCodeListKeywords">
               <gmd:keyword>
                  <gco:CharacterString>meteorology</gco:CharacterString>
               </gmd:keyword>
               <gmd:type>
                  <gmd:MD_KeywordTypeCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#MD_KeywordTypeCode"
                                          codeListValue="theme"/>
               </gmd:type>
            </gmd:MD_Keywords>
         </gmd:descriptiveKeywords>
         <gmd:resourceConstraints>
            <gmd:MD_LegalConstraints>
               <gmd:useLimitation gco:nilReason="missing">
                   <gco:CharacterString />
               </gmd:useLimitation>
               <gmd:accessConstraints>
                  <gmd:MD_RestrictionCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#MD_RestrictionCode"
                                          codeListValue="otherRestrictions"/>
               </gmd:accessConstraints>
               <gmd:useConstraints>
                  <gmd:MD_RestrictionCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#MD_RestrictionCode"
                                          codeListValue="otherRestrictions"/>
               </gmd:useConstraints>
               <gmd:otherConstraints>
                  <gco:CharacterString>Non-commerical use only</gco:CharacterString>
               </gmd:otherConstraints>
            </gmd:MD_LegalConstraints>
         </gmd:resourceConstraints>
         <gmd:language>
            <gmd:LanguageCode codeList="http://www.isotc211.org/2005/resources/codeList.xml#LanguageCode"
                              codeListValue="eng"/>
         </gmd:language>
         <gmd:topicCategory>
            <gmd:MD_TopicCategoryCode>climatologyMeteorologyAtmosphere</gmd:MD_TopicCategoryCode>
         </gmd:topicCategory>
         <gmd:extent>
            <gmd:EX_Extent id="boundingExtent">
               <gmd:geographicElement>
                  <gmd:EX_GeographicBoundingBox>
                     <gmd:westBoundLongitude>
                        <gco:Decimal>-180.0</gco:Decimal>
                     </gmd:westBoundLongitude>
                     <gmd:eastBoundLongitude>
                        <gco:Decimal>180.0</gco:Decimal>
                     </gmd:eastBoundLongitude>
                     <gmd:southBoundLatitude>
                        <gco:Decimal>-90.0</gco:Decimal>
                     </gmd:southBoundLatitude>
                     <gmd:northBoundLatitude>
                        <gco:Decimal>90.0</gco:Decimal>
                     </gmd:northBoundLatitude>
                  </gmd:EX_GeographicBoundingBox>
               </gmd:geographicElement>
            </gmd:EX_Extent>
         </gmd:extent>
      </gmd:MD_DataIdentification>
  </gmd:identificationInfo>
</gmd:MD_Metadata>

ywang-bom · 2014-12-02T04:39:19Z

Updated to clarify the nature of the bug as requested during 2014-11-28 OpenWIS TMC

Ticket title modified to better describe the issue
Ticket contents updated for clarification.
A test case is provided.

francoisfaugeron · 2014-12-15T10:03:56Z

Thanks for the clarification: we reproduced the bug.
After checking the issue, Remy and I think your possible fix might be a good solution.

gollogly · 2016-03-07T01:26:00Z

Hi Francois,

I have added the links to the binaries here which also hosts the latest jenkins builds

https://repository-openwis-association.forge.cloudbees.com/openwis/

gollogly · 2016-03-09T06:55:47Z

HI Dom

Please can you check the status of this issue recently revisited in the TC meeting. Please can you attach the metadata record that was used to produce the issue with non wmo data for further investigation by Francois

Thanks

woollattd · 2016-03-09T08:34:31Z

I used the same sample that Yang provided, but here it is as an attachment.
nonWMO.zip
(I've had to add it as a .zip as .xml is not supported in this window)

gollogly · 2016-03-09T09:18:47Z

Excellent, thanks Dom, I shall assign this back to Francois and let Remy know as well.

golfvert · 2016-03-14T16:40:10Z

I am confused by the term nonWMO. If I understand correctly,this is a metadata which is not for GlobalExchange. OK. However, I believe that a metadata compliant with WCMP 1.3 must have a data policy attached WMOOther for example.As far as I know, this is a must. In the example sent by Dom, I can't see that. So, for me, it is not a WMO complient metadata hence, it is quite 'normal' that OpenWIS refuses to use that.
Am I missing something?
Or, what is a nonWMO metadata?

ywang-bom · 2016-03-15T00:08:05Z

@remygiraud The metadata in question is for local/regional data, i.e. not to be exchanged globally and not to be Cached. The WMO core metadata profile 1.3 requires the data policy/license code (i.e. WMOOther etc.) to be present only for global exchanged products (9.3.1). Since the metadata is NOT for global exchange (it could be RegionalExchange or OriginatingCentre or N/A), it is not required to have an associated policy/license code and is still a valid WCMP 1.3 metadata.

francoisfaugeron · 2016-03-15T12:38:11Z

the metadata urn:x-wmo:md:au.gov.bom::ticket-32-test-1 is rejected by OpenWIS portal configured as follows (default) in openwis-metadataportal.properties:

openwis.metadataportal.extract.xpath=gmd:identificationInfo/gmd:MD_DataIdentification/gmd:resourceConstraints/gmd:MD_LegalConstraints/gmd:otherConstraints/gco:CharacterString,\
        gmd:identificationInfo/gmd:MD_DataIdentification/gmd:resourceConstraints/gmd:MD_LegalConstraints/gmd:useLimitation/gco:CharacterString

the metadata urn:x-wmo:md:au.gov.bom::ticket-32-test-1 enters the catalog of a portal (with openwis 3.14.x and other versions) configured as follows in openwis-metadataportal.properties:

openwis.metadataportal.extract.xpath=gmd:identificationInfo/gmd:MD_DataIdentification/gmd:resourceConstraints/gmd:MD_LegalConstraints/gmd:useLimitation/*,\
        gmd:identificationInfo/gmd:MD_DataIdentification/gmd:resourceConstraints/gmd:MD_LegalConstraints/gmd:otherConstraints/*

You can see an example with this url: http://dcpc-vaac-int.meteo.fr/openwis-user-portal/srv/en/main.home?urn=urn:x-wmo:md:au.gov.bom::ticket-32-test-1
This configuration is a Meteo France choice (because of DCPCs, Inspire compliance, ...). I think that many others can work.

rogers492 · 2017-03-02T22:41:21Z

This issue was moved to OpenWIS/openwis4#40

woollattd · 2018-06-13T10:30:46Z

reopened since OpenWIS v3.x is in maintenance mode for next 6 years so this still applies

ywang-bom added the Bug label Nov 18, 2014

ywang-bom changed the title ~~Cannot create metadata due to GTS Category being set to an empty string~~ Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" Dec 2, 2014

gollogly assigned gollogly and francoisfaugeron and unassigned gollogly Mar 7, 2016

gollogly assigned woollattd and unassigned francoisfaugeron Mar 9, 2016

gollogly assigned francoisfaugeron and unassigned woollattd Mar 9, 2016

ywang-bom unassigned francoisfaugeron Apr 6, 2016

rogers492 added the Discovery label Feb 9, 2017

lmika-bom modified the milestones: v3-only, v4, v3 Feb 21, 2017

lmika-bom added this to the v4 milestone Mar 2, 2017

rogers492 mentioned this issue Mar 2, 2017

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" OpenWIS/openwis4#40

Closed

rogers492 closed this as completed Mar 2, 2017

woollattd reopened this Jun 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" #32

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" #32

ywang-bom commented Nov 18, 2014

ywang-bom commented Dec 2, 2014

ywang-bom commented Dec 2, 2014

francoisfaugeron commented Dec 15, 2014

gollogly commented Mar 7, 2016

gollogly commented Mar 9, 2016

woollattd commented Mar 9, 2016

gollogly commented Mar 9, 2016

golfvert commented Mar 14, 2016

ywang-bom commented Mar 15, 2016

francoisfaugeron commented Mar 15, 2016

rogers492 commented Mar 2, 2017

woollattd commented Jun 13, 2018

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" #32

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category" #32

Comments

ywang-bom commented Nov 18, 2014

ywang-bom commented Dec 2, 2014

ywang-bom commented Dec 2, 2014

francoisfaugeron commented Dec 15, 2014

gollogly commented Mar 7, 2016

gollogly commented Mar 9, 2016

woollattd commented Mar 9, 2016

gollogly commented Mar 9, 2016

golfvert commented Mar 14, 2016

ywang-bom commented Mar 15, 2016

francoisfaugeron commented Mar 15, 2016

rogers492 commented Mar 2, 2017

woollattd commented Jun 13, 2018