Add ability to describe published data according to standard schema

[paulmillar]

Add ability to describe published data according to standard schema

Summary

SciCat has the concept of published data; that is, a set of one or more datasets that, collectively, are described by certain metadata fields. This metadata description is stored as a MongoDB document with the class PublishedData.

The backend has the ability to map this information to DataCite's XML schema, but only does this when making DataCite API requests for DOI activity. This ability to map PublishedData to a corresponding DataCite XML description isn't exposed by a SciCat API.

Perhaps because of this lack of exposing the DataCite description, the oai-provider-service reimplements the same mapping functionality (albeit not completely consistently). OAI-PMH also provides a Dublin Core description (as require by the OAI-PMH specification), which might also be useful under different circumstances.

Steps to Reproduce

When minting a DOI, SciCat backend generates XML that conforms to DataCite XML schema. OAI-PMH does the same, when querying the OpenAIRE (/openaire/oai) OAI-PMH endpoint.

Current Behaviour

Any client that wishes to generate a standards-compliant description of a PublishedData document needs to implement the mapping itself. This implies duplication of effort.

Should the PublishedData class be extended, so additional metadata is recorded (e.g., ORCIDs) and that additional metadata can be included in some standard metadata description (e.g., DataCite) then all service that generate that metadata description would need to be updated (e.g., DOI minting, OAI-PMH).

Expected Behaviour

The PublishedData API endpoint is extended to support querying for a description of a specific PublishedData document. This API extension would likely take two arguments: the metadata standard (e.g., Dublin Core, DataCite, Schema.org, ...) and the serialisation. In some cases, only one serialisation makes sense (e.g., DataCite and XML), but in other cases there may be multiple possible serialisations (e.g., Schema.org as JSON-LD,Turtle, RDF/XML, N3 ...).

The backend DOI minting activity would take advantage of this ability (although it might not issue HTTP requests) when generating the XML metadata for DataCite. The OAI-PMH interface could talk with the backend, rather than querying the MongoDB directly. The landing page could take advantage of this when including a Schema.org/JSON-LD description of the published data.

These would be natural places where this new API could be used (there may be others). I suggest this issue is closed when the extended PublicData API is available; ancillary issues should be opened against other SciCat components to track progress in adopting the new API (as appropriate).

Extra Details

This issue is the result of discussion on issue #1175. Some of the comments there are useful for this issue.

[paulmillar]

To provide more precise pointers:

• the XML metadata for DataCite is (currently) generated in the scicat-backend-next, towards the end of published-data.controller.ts.
• the OAI-PMH DataCite description of a record is provided in oai-provider-service repo, in openaire-mapper.ts

As an example of inconsistencies:

concept	published-data.controller.ts	openaire-mapper.ts
description[@type=Abstract]	abstract	dataDescription
creator/{givenName, familyName}	split on first space in name; first term is givenName, rest is familyName	These fields are not provided.
creator/creatorName	familyName, givenName (e.g., Millar, Paul)	creator (e.g., Paul Millar)
resourceType[@Dataset]	resourceType	This field is not provided.

[paulmillar]

To make this more actionable, here are two concrete proposals on how to add this ability.

1. Using content-negotiation.

In Section 5.3, RFC 7231 describes content negotiation as "proactive negotiation of the response content". The client sends Accept header, indicating the kind (as is Media Type) of response it can handle. The server can use this to choose the most appropriate response, if multiple formats are supported.

Based on this standard, the API endpoint /api/v3/publisheddata/{id} is extended so that it may return different descriptions of the published data entry with the specified ID. The default would be backwards-compatible (JSON) description, but the client can choose a different description based on some Media Type; e.g., to receive the DataCite XML version, send a GET request with the header:

Accept: application/vnd.datacite.datacite+xml

[As an aside, three DOI registrants (Crossref, DataCite and mEDRA) have agreed to use HTTP Content Negotiation as a common approach to providing DOI-base metadata.]

There seems to be some support in NestJS for Content-Negotiation, with a worked example. This information may be out-of-date and NestJS may now have better support.

2. Extending API

A relatively simple approach is to create additional endpoints, one for each format; for example, one could image the following additional endpoints:

• /api/v3/publisheddata/{id}/info/datacite+xml GET -- returns description in DataCite XML
• /api/v3/publisheddata/{id}/info/datacite+json GET -- returns description in DataCite JSON
• /api/v3/publisheddata/{id}/info/dublincore+xml GET -- returns description in Dublin Core.

Comparison

Content negotiation is a well-established standard feature of HTTP that wouldn't require any changes to the backend REST API (i.e., no new endpoints). According to above the above resources, it is possible to implement content negotiation with NestJS.

Content negotiation isn't compatible with static HTML's a element, because there is no standard way to control the Accept header within the a-element. Note that this is not a problem when using JavaScript (c.f. frontend's and LandingPageServer's use of Angular).

The extending-the-API approach would require a new endpoint for each new description format.

These are not mutually exclusive: both content-negotiation and extending the API could be employed. For example, the above crosscite uses both, where the DOI resolver supports content negotiation, but individual registrars support metadata by extending their API.

Conclusion

My naive preference would be to go for content negotiation and hold back from extending the API until it's needed, but someone with more hands-on experience with NestJS might have a different opinion.

[nitrosx]

@paulmillar thank you so much for opening this issues and the well thought content.
I will bring to the attention of the collaborators, and collect the feedback here.
I see a lot of potential in your idea.

[paulmillar]

Thanks @nitrosx for your support here!