TG2-AMENDMENT_EVENTDATE_STANDARDIZED #61

iDigBioBot · 2018-01-05T15:47:43Z

TestField	Value
GUID	718dfc3c-cb52-4fca-b8e2-0e722f375da7
Label	AMENDMENT_EVENTDATE_STANDARDIZED
Description	Propose amendment of the value of dwc:eventDate to a valid ISO date.
TestType	Amendment
Darwin Core Class	Event
Information Elements ActedUpon	dwc:eventDate
Information Elements Consulted
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED if the value of dwc:eventDate was not a properly formatted ISO 8601-1 date but was unambiguous, and was altered to be a valid ISO 8601-1 date; otherwise NOT_AMENDED
Data Quality Dimension	Conformance
Term-Actions	EVENTDATE_STANDARDIZED
Parameter(s)
Source Authority
Specification Last Updated	2023-09-18
Examples	[dwc:eventDate="2021-28-10": Response.status=AMENDED, Response.result=dwc:eventDate="2021-10-28", Response.comment="dwc:eventDate contains an interpretable value. Assuming year-day-month input format"]
	[dwc:eventDate="10-28": Response.status=NOT_AMENDED, Response.result=, Response.comment="dwc:eventDate contains an ambiguous value"]
Source	Paul Morris, Lee Belbin
References	ISO 8601-1: Latest version (currently 2019(en)) Date and time — Representations for information interchange — Part 1: Basic rules (https://www.iso.org/obp/ui/#iso:std:iso:8601:-1:ed-1:v1:en) Wikipedia (2020). ISO 8601 (https://en.wikipedia.org/wiki/ISO_8601)
Example Implementations (Mechanisms)	FilteredPush/Kurator:event_date_qc 10.5281/zenodo.596795.
Link to Specification Source Code	event_date_qc DwCEventDQ.amendmentEventdateStandardized() A minimal set of unit tests is in DwCEventDQTestDefinitions unit tests for the underlying verbatim date extraction code are in DateUtilsTest and DateUtilsTest
Notes	The intent of the amended range is to capture the original uncertainty where possible. As in the example, we amend "1999-11" instead of "1999-11-01/1999-11-31". An AMBIGUOUS response is possible.

iDigBioBot · 2018-01-05T15:47:44Z

Comment by Lee Belbin (@Tasilee) migrated from spreadsheet:
Response to PJM comments (#6a) on dateModified equivalent for eventDate - and scores copied from 6a for consistency (no gaps)

chicoreus · 2018-02-01T15:05:24Z

event_date_qc has a test with a guid 134c7b4f-1261-41ec-acb5-69cd4bc8556f referencing EVENTDATE_FORMAT_CORRECTION, need to confirm the correct GUID for this test.

chicoreus · 2018-02-01T15:08:38Z

I don't agree with the note. "1999-11" and "1999-11-01/1999-11-31" are both valid ISO formatted dates representing the interval of time of November 1999. They have identical meanings, and are not different representations of precision (at least as far as I've been able to tell from the publicly available information on the ISO standard.

chicoreus · 2018-02-01T15:23:05Z

Specification needs to describe how to handle ambiguity. 02/03/1884 is an ambiguous date and result should include a result state of AMBIGUOUS when providing an ISO format for such values, or the specification should provide limits on what forms of date are to be conformed with the ISO standard.
For maximum utility (and consistency), I would recommend that this test support the same set of date recognition and translation to ISO format as #86 TG2-AMENDMENT_EVENTDATE_FROM_VERBATIM, except for taking the value of eventDate as input.

ArthurChapman · 2018-02-01T22:10:25Z

As in the note I left at #86 to change the Tests Prerequisites to read ""The field dwc:eventDate is not EMPTY and is unambiguously interpretable as an ISO 8601:2004(E) date"

Tasilee · 2018-02-02T03:12:31Z

I agree with @ArthurChapman and again note we need to consider responses from tests (as note in #86)

…dwg/bdq#84 tdwg/bdq#26 tdwg/bdq#130 tdwg/bdq#61 PURPOSE: Updating tests to reflect updates in definitions and latest discussion in tdwg/bdq issues. DESCRIPTION: Removing confirmed unused isMonthInRange() method. Updating VALIDATION_STARTDAYOFYEAR_OUTOFRANGE to match new specification. Switching from #141 to #84 to test year for valid range. Correcting handling of ambiguity in AMENDMENT_EVENTDATE_STANDARDIZED and AMENDMENT_DATEIDENTIFIED_STANDARDIZED to conform with specifications.

…and added unit tests resulting from Arthur Chapman's ongoing review of results of test runs. DESCRIPTION: Set distinct guid for mechanism for DwCEventDateTG2DQ. Java to 1.8 and added commons-lang3 in pom for StringUtils. Moved implementation for dateIdentified into DwCOtherDateDQ. Fix for single digit days and months being recognized as valid ISO date, updates to unit tests in consequence. DateUtils.extractInterval() and extractDate() now returns null when given single digit day or month values. Fix for handling of date ranges with end date before start date. DateUtils.eventDateValid() now returns false on these. added unit tests for validationYearEmpty, amendmentDayStandardized, amendmentEventdateStandardized, validationDateidentifiedNotstandard.

… specifications. DESCRIPTION: Updating implementation and fixing unit test for AMENDMENT_EVENTDATE_STANDARDIZED to conform with current (2022-03-10) specification.

… specifications. DESCRIPTION: Fixing error in unit tests and in implementation of AMENDMENT_EVENTDATE_STANDARDIZED current specification is that internal prerequisites not met is only returned for an empty eventDate, but this was returned for other cases, as detected by running against the validation data.

Tasilee · 2022-03-30T01:33:12Z

"Original was "unambiguously conform..." while I think it should read

"AMENDED if the value of dwc:eventDate was changed to conform to an unambiguous valid ISO 8601-1:2019 date.."

chicoreus · 2022-03-30T01:42:52Z

I think the formulation "unambiguously conform" is correct, it tells the implementer that the act of making the data conform needs to unambiguously produce a single result, while the "unambiguous valid" simply reads as redundant for "valid", and and is saying that the result must unambiguously be a valid date, rather than saying that if there is the potential for more than one valid result, that path can't be taken.

ArthurChapman · 2022-03-30T01:45:53Z

An ISO value can't be ambiguous can it? I think "unambiguously conform..." is OK

Tasilee · 2022-03-30T02:02:04Z

OK, done

Tasilee · 2023-03-10T01:59:27Z

Due to recent discussions, I have changed

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it can be unambiguously conformed to bdq:sourceAuthority; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it can be unambiguously conformed to ISO 8601-1:2019; otherwise NOT_AMENDED

...and removed the reference to bdq:sourceAuthority

tucotuco · 2023-03-10T18:10:25Z

Again picky, but I would say:

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it can be unambiguously transformed to ISO 8601-1:2019; otherwise NOT_AMENDED

Tasilee · 2023-03-11T01:35:48Z

Edited accordingly

Tasilee · 2023-03-26T22:09:12Z

I've edited the Expected Response according to @tucotuco suggestion:

From

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it can be unambiguously transformed to ISO 8601-1:2019; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED |

and updated the References

ArthurChapman · 2023-03-27T22:38:17Z

See my comment under #26 Would the following be better?

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was unambiguous by formatting as a valid ISO 8601-1 date; otherwise NOT_AMENDED

Tasilee · 2023-03-29T22:10:39Z

I have updated the Expected Response as suggested above and the ISO Reference.

ArthurChapman · 2023-03-29T22:46:46Z

I think the Expected Response should be similar to #26 i.e.

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

chicoreus · 2023-06-10T16:58:55Z

Like #26 the language has become unclear for implementors: "AMENDED the value of dwc:eventDate if it was unambiguous and formatted as a valid ISO 8601-1 date" is read as formatted meaning "was already in the form", not meaning that it was changed. We need to revert to language from a prior version that is explicit about AMENDED being when the data are changed.

Propose changing from:

"INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED"

To:

"INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was not a properly formatted ISO 8601-1 date but was unambiguous and was altered to be a valid ISO 8601-1 date; otherwise NOT_AMENDED"

chicoreus · 2023-06-10T17:08:18Z

Both here and in #26 we want to amend if:

the existing value was not a correctly formatted data
the existing value was unambiguous (e.g. not a 2 digit year, not nn-nn-yyyy where nn are both below 12).
the value could be transformed into an ISO date.

…6-09) test descriptions. Adding ProvidesVersion annotations. Removing now empty file stubs for checked methods. Removed deprecated wrapper for method. Addressed tdwg/bdq#61 AMENDMENT_EVENTDATE_STANDARDIZED. Noted that like tdwg/bdq#26 the specification needs work to clarify the intent.

Tasilee · 2023-06-11T01:08:34Z

Again, I think the Expected Response may be clearer if your

"INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was not a properly formatted ISO 8601-1 date but was unambiguous and was altered to be a valid ISO 8601-1 date; otherwise NOT_AMENDED"

is rendered as

"INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was not a properly formatted ISO 8601-1 date but was unambiguously interpreted as a valid ISO 8601-1 date; otherwise NOT_AMENDED"

Tasilee · 2023-06-12T22:45:13Z

In line with #26, altered Expected response from

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED the value of dwc:eventDate if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY; AMENDED if the value of dwc:eventDate was not a properly formatted ISO 8601-1 date but was unambiguous, and was altered to be a valid ISO 8601-1 date; otherwise NOT_AMENDED

…specifications (2023-06-23). Updating implementation of tdwg/bdq#61 AMENDMENT_EVENTDATE_STANDARDIZED and tdwg/bdq#26 specification and metadata changed to be consistent with current behavior of code.

Tasilee · 2023-09-18T00:43:56Z

Splitting bdqffdq:Information Elements into "Information Elements ActedUpon" and "Information Elements Consulted".

Also changed "Field" to "TestField", "Output Type" to "TestType" and updated "Specification Last Updated"

iDigBioBot added Amendment TG2 labels Jan 5, 2018

ArthurChapman added the TIME label Jan 7, 2018

ArthurChapman added VOCABULARY Test Tests created by TG2, either CORE, Supplementary or DO NOT IMPLEMENT and removed VOCABULARY labels Jan 17, 2018

iDigBioBot added the Conformance label Jan 26, 2018

Tasilee mentioned this issue Jan 28, 2018

TG2-Remove all prerequisites for validations referencing amendments. #138

Closed

ArthurChapman added the ISO/DCMI STANDARD label Jan 29, 2018

chicoreus added the NEEDS WORK label Feb 1, 2018

Tasilee mentioned this issue Feb 11, 2018

TG2-VALIDATION_EVENTDATE_INRANGE #36

Open

ArthurChapman mentioned this issue Mar 21, 2018

TG2-VALIDATION_EVENTDATE_NOTEMPTY #33

Open

Tasilee removed the NEEDS WORK label Mar 23, 2018

Tasilee mentioned this issue Aug 14, 2018

suggest test for date format consistency and suggested revision #143

Closed

ArthurChapman added this to TIME TESTS in Core Tests and Assertions (TG2) Aug 27, 2018

Tasilee mentioned this issue Sep 3, 2018

TG2-AMENDMENT_DCTYPE_STANDARDIZED #41

Open

ArthurChapman added the responseDone label Nov 1, 2018

Tasilee removed the responseDone label Nov 5, 2018

chicoreus mentioned this issue Aug 14, 2019

TG2-AMENDMENT_DATEIDENTIFIED_STANDARDIZED #26

Open

Tasilee mentioned this issue Apr 13, 2020

TG2-AMENDMENT_[]_STANDARDIZED source authorities #185

Closed

ymgan mentioned this issue Feb 15, 2023

eventDate, year, month, day checks iobis/obis-qc#24

Open

17 tasks

chicoreus added the NEEDS WORK label Jun 10, 2023

Tasilee removed the NEEDS WORK label Jun 12, 2023

chicoreus added the CODED label Jul 4, 2023

chicoreus added the CORE TG2 CORE tests label Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TG2-AMENDMENT_EVENTDATE_STANDARDIZED #61

TG2-AMENDMENT_EVENTDATE_STANDARDIZED #61

iDigBioBot commented Jan 5, 2018 •

edited by Tasilee

iDigBioBot commented Jan 5, 2018

chicoreus commented Feb 1, 2018

chicoreus commented Feb 1, 2018

chicoreus commented Feb 1, 2018

ArthurChapman commented Feb 1, 2018

Tasilee commented Feb 2, 2018

Tasilee commented Mar 30, 2022 •

edited

chicoreus commented Mar 30, 2022

ArthurChapman commented Mar 30, 2022

Tasilee commented Mar 30, 2022

Tasilee commented Mar 10, 2023

tucotuco commented Mar 10, 2023

Tasilee commented Mar 11, 2023

Tasilee commented Mar 26, 2023

ArthurChapman commented Mar 27, 2023

Tasilee commented Mar 29, 2023

ArthurChapman commented Mar 29, 2023

chicoreus commented Jun 10, 2023

chicoreus commented Jun 10, 2023 •

edited

Tasilee commented Jun 11, 2023

Tasilee commented Jun 12, 2023

Tasilee commented Sep 18, 2023

TG2-AMENDMENT_EVENTDATE_STANDARDIZED #61

TG2-AMENDMENT_EVENTDATE_STANDARDIZED #61

Comments

iDigBioBot commented Jan 5, 2018 • edited by Tasilee

iDigBioBot commented Jan 5, 2018

chicoreus commented Feb 1, 2018

chicoreus commented Feb 1, 2018

chicoreus commented Feb 1, 2018

ArthurChapman commented Feb 1, 2018

Tasilee commented Feb 2, 2018

Tasilee commented Mar 30, 2022 • edited

chicoreus commented Mar 30, 2022

ArthurChapman commented Mar 30, 2022

Tasilee commented Mar 30, 2022

Tasilee commented Mar 10, 2023

tucotuco commented Mar 10, 2023

Tasilee commented Mar 11, 2023

Tasilee commented Mar 26, 2023

ArthurChapman commented Mar 27, 2023

Tasilee commented Mar 29, 2023

ArthurChapman commented Mar 29, 2023

chicoreus commented Jun 10, 2023

chicoreus commented Jun 10, 2023 • edited

Tasilee commented Jun 11, 2023

Tasilee commented Jun 12, 2023

Tasilee commented Sep 18, 2023

iDigBioBot commented Jan 5, 2018 •

edited by Tasilee

Tasilee commented Mar 30, 2022 •

edited

chicoreus commented Jun 10, 2023 •

edited