Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting

[philippconzett]

Note: dc:rights is being handled in #5920 and #4176 but the original description of this issue has been preserved.

Based on a semi-systematic survey of how DataverseNO metadata is harvested in Bielefeld Academic Search Engine (BASE; https://www.base-search.net/Search/Advanced), a major search engine for research outputs, we have noticed some issues related to the way the Dataverse software provides Dublin Core metadata for OAI-PMH harvesting.

dc:type
BASE harvests multiple types of research output, e.g. publications and datasets. Searching BASE you can filter/limit the search result to only include datasets by selecting Dataset in the Document Type section of advanced search:

However, only very few metadata records harvested directly from DataverseNO are marked as Document Type = Dataset.
It seems that in the oai_dc format, which BASE uses for harvesting, Document Type is based on the dc:type field. According to the Dataverse Metadata Crosswalk, dc:type corresponds to the Dataverse metadata field Kind of Data. But this field may contain very different values, e.g., “survey data”, “survey”, “observations” etc. Dublin Core (see https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#http://purl.org/dc/terms/type) recommends “to use a controlled vocabulary such as the DCMI Type Vocabulary” for dc:type. The DCMI Type Vocabulary has “dataset” as one of its values. I therefore suggest changing the Dataverse / DC Element (oai_dc) mapping, so that dc:type is hard-coded as “dataset” for all dataset metadata in Dataverse.

dc:date
The Dataverse metadata field Publication Date is available as dcterms:issued, but it doesn’t seem to be among the oai_dc fields Dataverse exposes for OAI-PMH harvesting. According to the Dataverse Metadata Crosswalk, dc:date corresponds to the Dataverse metadata field Deposit Date, but all the random samples I tested in BASE indicate that dc:date, which BASE uses as input for their metadata field Year of Publication, corresponds to the Dataverse field Date of Production. I suggest changing the Dataverse / DC Element (oai_dc) mapping, so that dc:date is mapped with Publication Date. This is also in line with citation recommendations. The publication date is the preferred date when citing research data; see, e.g., page 12 in The Tromsø Recommendations for Citation of Research Data in Linguistics; https://doi.org/10.15497/rda00040.

dc:rights
For some of the sources included in BASE, there is an indication of the degree of Open Access. Among them are some Dataverse-based repositories. On the other side, for DataverseNO and other Dataverse-based repositories, this information is not available / unknown (“unbekannt”):

The Open Access information in BASE is based on the Dublin Core field dc:rights. Dataverse does not provide the field dc:rights. A correct value in this field would enable BASE to indicate the degree of Open Access (see more information at https://www.base-search.net/about/en/faq_oai.php#dc-rights). For datasets without access restriction, the dc:rights field could look like this: info:eu-repo/semantics/openAccess (see more information at https://guidelines.openaire.eu/en/latest/data/field_rights.html#rightsuri-ma).

[poikilotherm]

Related:

• Usage of datestamp from Record during harvest using oai_dc format #8693

[pdurbin]

I suggest changing the Dataverse / DC Element (oai_dc) mapping, so that dc:date is mapped with Publication Date.

I believe that @tcoupin fixed this in the following pull request, which we just merged and will be available in the next version of Dataverse (5.13 as of this writing):

• oai_dc export: use publication date if production date is not set #8733

By the way, thank you @philippconzett for the extensive write up! It's a lot to go through. Very thorough. 😄

[jggautier]

Related:

• Access Rights metadata in OpenAIRE metadata export is being misapplied #5920

[cmbz]

2024/05/08

• Identify which elements of the request have already been addressed by open or closed issues
• Split remaining elements into their own issues
• Then close this issue

[DS-INRAE]

Another related issue:

• As a data repository, I need to harvest additional metadata in OAI_DC records #4176

[pdurbin]

@philippconzett (and any others watching this issue), I create a pull request to address the points you made above:

• Remap oai_dc fields dc:type and dc:date #10737

Please take a look and feel free to leave comments or a review on the pull request. Thanks.

[philippconzett]

@pdurbin Thanks! I just left a comment on the PR.

[pdurbin]

This issue was just closed because we merged the following pull request:

• Remap oai_dc fields dc:type and dc:date #10737

As explained above, changes to dc:rights were not included in the scope of the pull request. Please look instead to these issues:

• Access Rights metadata in OpenAIRE metadata export is being misapplied #5920
• As a data repository, I need to harvest additional metadata in OAI_DC records #4176