TG2-AMENDMENT_COUNTRYCODE_FROM_COORDINATES

[iDigBioBot]

TestField	Value
GUID	8c5fe9c9-4ba9-49ef-b15a-9ccd0424e6ae
Label	AMENDMENT_COUNTRYCODE_FROM_COORDINATES
Description	Proposes an amendment to the value of dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code.
TestType	Amendment
Darwin Core Class	dcterms:Location
Information Elements ActedUpon	dwc:countryCode
	dwc:decimalLatitude
	dwc:decimalLongitude
Information Elements Consulted
Expected Response	EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY, or if dwc:countryCode is NOT_EMPTY; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.
Data Quality Dimension	Completeness
Term-Actions	COUNTRYCODE_FROM_COORDINATES
Parameter(s)	bdq:sourceAuthority
Source Authority	bdq:sourceAuthority default = "ADM1 boundaries spatial UNION with Exclusive Economic Zones" {[https://gadm.org] spatial UNION [https://marineregions.org]}
Specification Last Updated	2024-08-18
Examples	[dwc:decimalLatitude="-25.23", dwc:decimalLongitude="135.43", dwc:countryCode="": Response.status=FILLED_IN, Response.result=dwc:countryCode="AU", Response.comment="dwc:decimalLatitude and dwc:decimalLongitude contain interpretable values"]
	[dwc:decimalLatitude="-38.280937", dwc:decimalLongitude="72.047790", dwc:countryCode="": Response.status=NOT_AMENDED, Response.result="", Response.comment="Coordinates do not fall in the boundary of any country"]
Source	ALA, GBIF, iDigBio
References	Chapman AD and Wieczorek JR (2020) Georeferencing Best Practices. Copenhagen: GBIF Secretariat. https://doi.org/10.15468/doc-gg7h-s853 VLIZ (2023). Marineregions.org. https://www.marineregions.org/downloads.php#marbound Dooley, JF Jnr. (2005) An inventory and comparison of globally consistent geospatial databases and libraries. Rome: FAO. http://www.fao.org/3/a0118e/a0118e00.htm#Contents Wikipedia (2020) ISO 3166-1 alpha-2. https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 DataHub (2018) List of all countries with their two digit codes (ISO 3166-1). https://datahub.io/core/country-list
Example Implementations (Mechanisms)
Link to Specification Source Code
Notes	This amendment simply fills dwc:countryCode from a lookup of dwc:decimalLatitude and dwc:decimalLongitude. dwc:coordinateUncertaintyInMeters and dwc:coordinatePrecicision (if present) imply a buffer around the provided coordinates. Likewise, country polygons cannot be 100% accurate at all scales (Dooley 2005), so a spatial buffer of the country boundaries is also justified. Taking spatial buffers into account does however greatly complicate the logic and the implementation of this and related tests. In this test, a detection of multiple country codes by sampling within the buffer while possible, is not considered.

[iDigBioBot]

Comment by Paula Zermoglio (@pzermoglio) migrated from spreadsheet:
Only useful if performed AFTER decimalLat and decimalLong interpretation.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
@pz amendment sequence is indeed important. Tianhong Song has a paper on this in the context of prerequisites for workflow actors.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Implementation requires guidance on how to handle marine material inside a country's exclusive economic zone.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Should include coordinateUncertainty in meters. Also see Lee's note in Principles about some geographic tests needing buffers

[ArthurChapman]

Change Country to Country Codes - name and elsewhere

[tucotuco]

Agreed at TDWG 2018 DQIG meeting that the name TG2-AMENDMENT_COUNTRYCODE_FROM_COORDINATES is satisfactory.

[tucotuco]

Similar to the problem raised in Issue #185, this tests mentions a source authority that can not deliver the AMENDMENT. It is also silent on the authority for the geometries for the country codes. To me, the bdq:sourceAuthority should be the GBIF reverse geocoding API (https://github.com/gbif/geocode), coming in 2020. It will be based on Natural Earth, GADM, Open Street Maps, EEZones and more. The documentation says it will be for internal GBIF use, but @timrobertson100 says that he expects the API to be exposed.

[timrobertson100]

The geocode service is available today, and provides a lookup based on the given coordinate.

For example latitude 51.0 and longitude 1.0 yields this response:

[
  {
    "id": "80",
    "type": "Political",
    "source": "http://www.naturalearthdata.com",
    "title": "United Kingdom",
    "isoCountryCode2Digit": "GB"
},
{
    "id": "212",
    "type": "EEZ",
    "source": "http://vliz.be/vmdcdata/marbound/",
    "title": "United Kingdom",
    "isoCountryCode2Digit": "GB"
    }
]

Because boundaries don't align (resolution of the polygons), we buffer the search which is why multiple results can be returned.

To reduce WS traffic, we also encode the database into an image with dictionary encoded colors (i.e. by seeing a non-black or white colour, you can refer to the dictionary to know the country).

Today the service only has EEZ and NaturalEarth files, but can (will) be extended.

When a record has a stated country and coordinates we verify that seems reasonable, and if not flips coordinates around and negates them "hunting" for a match. This is because in many cases the negative sign is omitted, or coordinates swapped. All of this happens after a reprojection to WGS84 if necessary.

[tucotuco]

@timrobertson100 This is ideal. May we cite it as our bdq:sourceAuthority default?

[chicoreus]

We are conflating authority with service with thesaurus in bdq:sourceAuthority. The authority for country codes is the ISO two letter country code list. The GBIF service is a service that wraps natural earth plus (source?) EEZ layers (and will change overtime as layers are added (with versioning?), the thesaurus is the natural earth and EEZ layers. For implementation, I'd much rather use a local GIS data store containing natural earth data and some appropriate EEZ layer than consulting a remote service - I want to use the same thesaurus, but not the service.

[tucotuco]

OK, what is the consensus then on what to cite and how? I think it is more useful to point people to the thesaurus. If that can be done in a general way instead of to a particular service built around it, so much the better. For example, is https://github.com/gbif/geocode the right thing to cite for the source authority for this test? What does not seem useful is to just give the controlled vocabulary that the amendment should comply with as the source authority, and to not have the thesaurus in the References. The controlled vocabulary is incomplete as an authority for amendments of this type.

…

On Tue, Apr 14, 2020 at 11:51 AM Paul J. Morris ***@***.***> wrote: We are conflating authority with service with thesaurus in bdq:sourceAuthority. The authority for country codes is the ISO two letter country code list. The GBIF service is a service that wraps natural earth plus (source?) EEZ layers (and will change overtime as layers are added (with versioning?), the thesaurus is the natural earth and EEZ layers. For implementation, I'd much rather use a local GIS data store containing natural earth data and some appropriate EEZ layer than consulting a remote service - I want to use the same thesaurus, but not the service. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADQ72YD26BR3XCMM6AMOULRMRZ63ANCNFSM4EKSNA7Q> .

[Tasilee]

Thanks @timrobertson100 , @tucotuco and @chicoreus. I am the least able person in the group to provide wisdom but figure I would put my thoughts down, anyway.

As we have discussed, GBIF is likely for the near future, to be a service location that aggregates thesauri and what we currently call 'bdq:sourceAuthority' (authorities).

Are the thesauri themselves 'source authorities'? If we are dependent on them as the reference, then in our context, I'd say yes. I agree with @chicoreus that the services associated with any source authority are a separate issue. That's why we use " if the bdq:sourceAuthority service ..."

To address @tucotuco 's question about how we reference, we either reference to the GBIF namespace end point (?) which references the relevant external source authority (e.g., ISO) in a standard way or we reference both the GBIF and the 'external' source authority. The former would be nice. Then there is a separate reference to the service associated with the thesaurus.

In the Expected response, we have agreed to use bdq:sourceAuthority [and service.] where there are implementation-dependent options or directly use the name of the source authority where there is no choice.

What we place in References and Notes, I defer to more appropriate authorities.

[ArthurChapman]

My view is same as @tuco and his question to @timrobertson100
"@timrobertson100 This is ideal. May we cite it as our bdq:sourceAuthority default"

If can just add the GBIF Geocode Authority as our bdq:sourceAuthority as we have done elsewhere, then I think this is our best option.

[chicoreus]

The more I think about this, the less I like the idea of specifying a query endpoint as a sourceAuthority. This leaves the resolution of the shape files, buffering near borders, and changes over time as opaque to the consumer, and also forces implementations to use a service for a test that should not be implemented with remote service calls but with a local spatial data store.

We should specify three parameters: (1) A shape file for country boundaries. (2) A shape file for exclusive economic zones. (3) An explicit buffer for points that fall near country boundaries that takes into account the resolution of the shapefiles. In addition, we should explicilty state in the specification how points that fall into the buffer are to be handled (preferably by not asserting an amendment, probably with INTERNAL_PREREQUISITES_NOT_MET, point falls too near boundary of shape to determine placement). In addition, we need to consider coordinatePrecision, and how that as an uncertainty on the coordinate intersects with countries or buffers.

We need to be much, much more explicit in how edge cases are to be handled in any test that involves GIS data.

This amendment also needs to cover the case of FILLED_IN, AMENDED would only cover an existing case of countryCode being altered based on the coordinates. We should consider if this test should ever assert AMENDED, or should restrict itself to FILLED_IN. I would generally be much more comfortable with asserting only FILLED_IN, as AMENDED could fix the wrong value (the coordinates, or their precision, or their error radius could be in error, but the country and countryCode be correct). This is particularly true near boundaries, where the error may lie in the resolution of the shape rather than either the coordinate or the countryCode.

[chicoreus]

I would suggest:

EXTERNAL_PREREQUISITES_NOT_MET if an external source authority service or local spatial data store was not available; INTERNAL_PREREQUISITES_NOT_MET if the fields dwc:decimalLatitude, dwc:decimalLongitude are EMPTY or the dwc:decimalLatitude and dwc:decimalLongitude cannot be converted to the SRS used for queries to the spatial service or data store, or if the area represented by the dwc:decimalLatitude, dwc:decimalLongitude, and dwc:coordinatePrecision overlaps with a 3km buffer zone on any country or EEZ shape in the service or spatial data store; FILLED_IN if the value of dwc:countryCode was EMPTY and was unambiguously inferred from supplied dwc:decimalLatitude, dwc:decimalLongitude and dwc:coordinatePrecision falling within a single boundary defined by the combination of terrestrial and exclusive economic zone and not overlapping a 3km buffer around such boundary; otherwise NOT_CHANGED

[ArthurChapman]

I would stay with AMENDED and NOT_AMENDED

[Tasilee]

Changed "AMENDED" to "FILLED_IN" in accordance with discussions April 16.

[Tasilee]

@ArthurChapman and I have discussed my recent changes to

• Expected Response (generic use of ISO 3166)
• Source Authority (added gadm and ISO)
• References (Do we duplicate the links to ISO and GADM?)

Also, this test no longer uses bdq:spatialBufferInMeters so the question needs to be asked if #50, #51 and #56. To be discussed at next Zoom. Also use of SRS? We need to be consistent.

[ArthurChapman]

We have dwc:coordinatePrecision and dwc:geodeticDatum in the Information Elements but they do not occur in the Expected Response?

Should these be deleted from the Information Elements or is our Expected Response deficient?

Note that dwc:coordinatePrecicision and dwc:coordinateUncertaintyInMeters occur in the Notes

[ArthurChapman]

in Expected Response bdq:sourceAuthority[countryShapes] changed to bdq:sourceAuthority[countryshapes] to conform with other similar terms under bdq:sourceAuthority

[Tasilee]

If we believe the Expected response is correct, then dwc:coordinatePrecision and dwc:geodeticDatum should not be information elements. Were we thinking a buffer?

[chicoreus]

The notes suggest using dwc:coordinatePrecision and dwc:geodeticDatum to specify a buffer, but also invoke the a buffer associated with the country boundary representation. But, we don't list a buffer as an element in the specification. So, in essence, we say both that a buffer should be used and that a buffer is not used...

[Tasilee]

Here are the tests that include buffer

#50

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if one or more of dwc:decimalLatitude, dwc:decimalLongitude, or dwc:countryCode are EMPTY or invalid; COMPLIANT if the geographic coordinates fall on or within the boundary defined by the union of the boundary of the country from dwc:countryCode plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferInMeters; otherwise NOT_COMPLIANT.

#51

EXTERNAL_PREREQUISITES_NOT_MET if either bdq:sourceAuthority[taxonomyismarine] or bdq:sourceAuthority[geospatialland] are not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:dcientificName was EMPTY or the marine/non-marine status of the taxon is not interpretable from bdq:sourceAuthority[taxonomyismarine] or the values of dwc:decimalLatitude or dwc:decimalLongitude are EMPTY; COMPLIANT if the taxon marine/non-marine status from bdq:sourceAuthority[taxonomyismarine] matches the marine/non-marine status of dwc:decimalLatitude and dwc:decimalLongitude on the boundaries given by bdq:sourceAuthority[geospatialland] plus an exterior buffer given by bdq:spatialBufferInMeters; otherwise NOT_COMPLIANT.

#56

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if the values of dwc:decimalLatitude, dwc:decimalLongitude, or dwc:stateProvince are EMPTY or invalid; COMPLIANT if the geographic coordinates fall on or within the boundary from the bdq:sourceAuthority for the given dwc:stateProvince (after coordinate reference system transformations, if any, have been accounted for), or within the distance given by bdq:spatialBufferInMeters outside that boundary; otherwise NOT_COMPLIANT.

BUT, I don't think a buffer is justified in this AMENDMENT. We are not comparing two things. We are looking up dwc:countryCode from coordinates. Using dwc:coordinateUncertaintyInMeters or dwc:coordinatePrecision or dwc:geodeticDatum wouldn't help, unless the test matches to country - which it doesn't of course.

So, I am going to amend the specs accordingly, and the rest of you can argue the case...

[Tasilee]

Was this the intent?

Notes

This amendment simply fills dwc:countryCode from a lookup of dwc:decimalLatitude and dwc:decimalLongitude. dwc:coordinateUncertaintyInMeters and dwc:coordinatePrecicision (if present) imply a buffer around the provided coordinates. Likewise, country polygons cannot be 100% accurate at all scales (Dooley 2005), so a spatial buffer of the country boundaries is also justified. Taking spatial buffers into account does however greatly complicate the logic and the implementation of this and related tests. In this test, a detection of multiple country codes by sampling within the buffer while possible, is not considered.

[ArthurChapman]

Another issue with this test (and the others where we have bdq:sourceAuthority[xxxx]). If these are separated into different terms in the Vocabulary as suggested as a possibility in #152 (comment) - then we would need to change Parameter(s) to bdq:sourceAuthority to "bdq:sourceAuthority[countryshapes]" and Source Authority to "bdq:sourceAuthority default = "ADM1 boundaries" [https://gadm.org] UNION with "EEZs" [https://marineregions.org]"

ALSO - we have in bdq:souceAuthority "bdq:sourceAuthority[countryCode]" but this term doesn't occur anywhere in the test (or any other test) and is not in the Vocabulary - should this be bdq:sourceAuthority[countryshapes]???

[Tasilee]

I have updated the Source authority and description

[ArthurChapman]

I don't think there is anything outstanding wrt NEEDS WORK - other than a decision on how we treat bdq:sourceAuthority[countryshapes]

[ArthurChapman]

Updated Expected Response, bdq:sourceAuthority and Description to replace bdq:sourceAuthority:[countryShapes] with bdq:countryShapes and Specification Last Updated

Changed Parameter(s) from bdq:sourceAuthority to bdq:countryShapes

[Tasilee]

I thought @tucotuco's suggestion was bdq:sourceAuthority_countryshapes (or maybe "countryShapes") etc? That way, we don't loose the source authority link.

[ArthurChapman]

We don't use that for any other bdq: terms But if all agree would could change it - but see the Vocab or #205 for all the terms we would have to change

| bdq:annotationAlertIf | [https://github.com//issues/152] | |
| bdq:countryShapes | [https://github.com//issues/152] | |
| bdq:annotationAnnotationSystem | [https://github.com//issues/152] | |
| bdq:defaultGeodeticDatum | [https://github.com//issues/152] | |
| bdq:earliestValidDate | [https://github.com//issues/152] | |
| bdq:geospatialLand | [https://github.com//issues/152] | |
| bdq:includeEventDate | [https://github.com//issues/152] | |
| bdq:latestValidDate | [https://github.com//issues/152] | |
| bdq:maximumValidDepthInMeters | [https://github.com//issues/152] | |
| bdq:maximumValidElevationInMeters | [https://github.com//issues/152] | |
| bdq:minimumValidDepthInMeters | [https://github.com//issues/152] | |
| bdq:minimumValidElevationInMeters | [https://github.com//issues/152] | |
| bdq:sourceAuthority | [https://github.com//issues/152] | |
| bdq:spatialBufferInMeters | [https://github.com//issues/152] | |
| bdq:targetCRS | [https://github.com//issues/152] | |
| bdq:taxonomyIsMarine | [https://github.com//issues/152] | |

[Tasilee]

Thanks @ArthurChapman. The list includes terms that do relate to a bdq:sourceAuthority (annotations, country shapes/land, geodetic datum) but the rest don't really, unless you want to stretch the concept.

I have therefore changed my mind (backflip?) and think that the syntax like "bdq:countryShapes" seems expedient as all bdqs are listed under the 'heading' "Source Authority" anyway.

[Tasilee]

Aligned the Source Authority entries between #50 and this test

[Tasilee]

Post Zoom 11/7/2023, I have aligned the Source Authority with the suggested syntax:

bdq:sourceAuthority default = "ADM1 boundaries" {[https://gadm.org] spatial UNION with "Exclusive Economic Zones" [https://marineregions.org]}

[Tasilee]

In the light of #50 suggestion, changed Source Authority to

| Source Authority | bdq:sourceAuthority default = "ADM1 boundaries spatial UNION with Exclusive Economic Zones" {[https://gadm.org] spatial UNION [https://marineregions.org]} |

[Tasilee]

Changed Parameter(s) from bdq:countryShapes to bdq:sourceAuthority and presume we will leave it Parameterized.

[Tasilee]

Splitting bdqffdq:Information Elements into "Information Elements ActedUpon" and "Information Elements Consulted".

Also changed "Field" to "TestField", "Output Type" to "TestType" and updated "Specification Last Updated"

[Tasilee]

Standardized reference to bdq:sourceAuthority in Expected Response to "EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available"

[Tasilee]

Changed Expected Response from

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.

to

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY, or if dwc:countryCode is NOT_EMPTY; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.