Source data validation issues

[strogonoff]

Describe the issue

Stefano outlined discrepancies between data sources and relaton-py expectations here.

The discrepancies arise because (1) previously invalid YAML was updated to match Relaton RNC/LutaML data specification (where relaton-py has come to rely on YAML to be invalid), or (2) data specification updated, YAML updated accordingly, but relaton-py was not updated.

These discrepancies lead to issues like: #277, #250, #281.

Fronts on which they could be attacked:

• Normalizing abnormal source data in BibXML service (fixes partly implemented in Normalize some source data variations #282)
• Making it so that abnormal source data doesn’t cause issues (not always realistic, but exploring options)
• Pinning Relaton gems by their minor version part in GHAs (and relying on gem maintainers to avoid introducing breaking data format changes within the same minor version; I was informed that’s how it already works)
• Updating Relaton specification to support versioning (medium term, we shouldn’t do it in rush)

Code of Conduct

• I agree to follow the IETF's Code of Conduct

[strogonoff]

As of now, the discrepancies are:

• IANA:

contributor -> 0 -> organization -> abbreviation
  str type expected (type=type_error.str) # fixed

• IEEE:

contributor -> 0 -> organization -> abbreviation
  str type expected (type=type_error.str) # fixed
keyword
  str type expected (type=type_error.str) # fixed
keyword -> 0
  str type expected (type=type_error.str) # fixed
copyright -> 0 -> owner -> 0 -> name -> 0
  str type expected (type=type_error.str)
copyright -> 0 -> owner -> 0 -> name
  str type expected (type=type_error.str)
copyright -> 0 -> owner -> 0 -> abbreviation
  str type expected (type=type_error.str) # fixed
copyright -> from
  field required (type=value_error.missing)
copyright -> owner
  value is not a valid list (type=type_error.list)

• ID:

contributor -> 0 -> person -> name -> given -> forename -> 0
  Forename.__init__() missing 1 required positional argument: 'content' (type=type_error)
contributor -> 0 -> person -> name -> given -> forename -> content
  str type expected (type=type_error.str)

• MISC:

contributor -> 0 -> person -> name -> given -> forename -> 0
  Forename.__init__() missing 1 required positional argument: 'content' (type=type_error)
contributor -> 0 -> person -> name -> given -> forename -> content
  str type expected (type=type_error.str)

• NIST:

contributor -> 8 -> organization -> abbreviation
  str type expected (type=type_error.str)

• RFCS:

contributor -> 0 -> person -> name -> given -> forename -> 0
  Forename.__init__() missing 1 required positional argument: 'content' (type=type_error)
contributor -> 0 -> person -> name -> given -> forename -> content
  str type expected (type=type_error.str)

• organization -> abbreviation cases should be covered by normalization in Normalize some source data variations #282
• person -> name -> given -> ... cases should be covered by normalization in Normalize some source data variations #282
• keyword -> ... cases should be covered by normalization in fix: normalize keywords (#277) #278
• copyright -> 0 -> owner -> ... cases still need attention (validation errors here: https://bib.ietf.org/indexed-sources/relaton-data-ieee/ANSI_IEEE_991-1986/)

[alicerusso]

adding some IEEE examples of this issue in practice. (presumably this is #283, although I don't see "Could not export" mentioned above.) Got the following errors upon click of "bibxml" on:

• https://bib.ietf.org/get-one/by-docid/?doctype=IEEE&docid=IEEE+1003.1-2017&format=bibxml
• https://bib.ietf.org/get-one/by-docid/?doctype=IEEE&docid=IEEE+1003.1-2016&format=bibxml
• https://bib.ietf.org/get-one/by-docid/?doctype=IEEE&docid=IEEE+802.1Q-2014&format=bibxml

Could not export this item, the error was: Source data for item IEEE 1003.1-2017 (IEEE) didn’t validate (err: 7 validation errors for BibliographicItem copyright -> 0 -> owner -> 0 -> name -> 0 str type expected (type=type_error.str) copyright -> 0 -> owner -> 0 -> name str type expected (type=type_error.str) copyright -> 0 -> owner -> 0 -> abbreviation str type expected (type=type_error.str) copyright -> 0 -> owner -> 1 -> name -> 0 str type expected (type=type_error.str) copyright -> 0 -> owner -> 1 -> name str type expected (type=type_error.str) copyright -> from value is not a valid integer (type=type_error.integer) copyright -> owner field required (type=value_error.missing))

the last one:

Could not export this item, the error was: Source data for item IEEE 802.1Q-2014 (IEEE) didn’t validate (err: 5 validation errors for BibliographicItem copyright -> 0 -> owner -> 0 -> name -> 0 str type expected (type=type_error.str) copyright -> 0 -> owner -> 0 -> name str type expected (type=type_error.str) copyright -> 0 -> owner -> 0 -> abbreviation str type expected (type=type_error.str) copyright -> from value is not a valid integer (type=type_error.integer) copyright -> owner field required (type=value_error.missing))

[ronaldtse]

The issues reported by @strogonoff have been moved here:

• (URGENT) Address structural discrepancies across Relaton data sources relaton/relaton#108

@alicerusso we will investigate the IEEE examples in:

• (URGENT) Structural discrepancies in Relaton-IEEE data files relaton/relaton-ieee#32

[ajeanmahoney]

Hi, @ronaldtse, is there an update for this issue? I see that the dependencies relaton/relaton#108 and relaton/relaton-ieee#32 are closed, and the reported discrepancies (#277, #250, #281.) are also closed. Thanks!

[rjsparks]

@ronaldtse is there anything left to do here?