Remap oai_dc fields dc:type, dc:date, and dc:rights

[pdurbin]

What this PR does / why we need it:

The oai_dc export and harvesting format has had the following fields remapped:

• dc:type was mapped to the field "Kind of Data". Now it is hard-coded to the word "Dataset".
• dc:date was mapped to the field "Production Date" when available and otherwise to "Publication Date". Now it is mapped only to the field "Publication Date".
• dc:rights was not mapped to anything. Now it is mapped (when available) to terms of use, restrictions, and license.

As these are backward incompatible changes, they have been noted in the API changelog: https://dataverse-guide--10737.org.readthedocs.build/en/10737/api/changelog.html

Which issue(s) this PR closes:

• Closes Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129

Special notes for your reviewer:

Should these backward-incompatible changes be hidden behind a feature flag?

Suggestions on how to test this:

See rules above under "what this PR does".

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No.

Is there a release notes update needed for this change?:

Yes, included.

Additional documentation:

Yes, I updated the API changelog: https://dataverse-guide--10737.org.readthedocs.build/en/10737/api/changelog.html

[pdurbin]

@jggautier heads up that this relates to this issue in that we are now adding "dc:rights" to the oai_dc export/harvesting format:

• Feature Request/Idea: Make sure that for datasets published with CC0 waiver or standard license, metadata exports include the waiver or license #8798

[pdurbin]

@tcoupin I'm requesting a review from you because you modified the dc:date login in the following pull request and I changed it (as explained above):

• oai_dc export: use publication date if production date is not set #8733

[qqmyers]

Re: dc:date - should it be mapped to the same field as https://guides.dataverse.org/en/latest/api/native-api.html#set-citation-date-field-type-for-a-dataset ? That is publicationDate by default.

[coveralls]

coverage: 20.735% (-0.006%) from 20.741%
when pulling c59f743 on 8129-harvesting
into 0d27957 on develop.

[pdurbin]

@qqmyers well, Publication Date is what @philippconzett asked for in the issue (#8129).

[qqmyers]

@philippconzett 's notes also point out that this date potentially going to be interpreted as the citation date. Since we allow configuring that in the local installation, it seems like it could be confusing to hardcode it for harvesting. If the harvester used the field from that setting, citations would be consistent in the local display and harvesting sites, and it would default to publicationDate as requested in the issue.

[pdurbin]

I don't have a strong opinion about it.

[philippconzett]

I think @qqmyers's suggestion for dc:date makes sense.

[plecor]

I've taken @tcoupin's role on Dataverse issues, so I am looking at this for him.

Part of the context for the change he implemented (mapping dc:date to Publication Date if Production Date is empty) was that when Dataverse harvests another OAI-PMH repo, dc:date is mapped to productionDate and this production date is then used in the citation of the harvested dataset. #8733 and #8732 were both part of an effort to guarantee the coherence between citation dates when harvesting another Dataverse.

So I agree with @qqmyers's suggestion on dc:date.

Hardcoding dc:type to Dataset would certainly simplify things. In practice, I know some Dataverse instances allow for kindOfData values that are not synonyms of 'Dataset'. For instance : https://entrepot.recherche.data.gouv.fr/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=doi:10.57745/DZIM2L

There might be an alternative solution where there is always at least a dc:type tag with value Dataset and the list of kindOfData values (making sure that Dataset occurs only once)? That would however means that not all dc:type values come from a controlled vocabulary.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:8129-harvesting

ghcr.io/gdcc/configbaker:8129-harvesting

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[pdurbin]

@plecor thanks. One thing to consider with "dc:type" is that types other than datasets (like software and workflows) are coming...

• dataset types (software, workflow, etc.) - initial support #10694

... so maybe we can revisit "dc:type" once that pull request is merged.

To all, I pushed some tests to exercise export and setting the citation date.

Now I'm trying to see if there's a small change I can make to DublinCoreExportUtil to get the citation date out.

I can get just the year (YYYY) with code like this...

String citation = version.getCitation();
// We're looking for ", YYYY, " in a citation like this:
// Finch, Fiona, 1999, "Darwin's Finches", https://doi.org/10.5072/FK2/WSSYBE, Root, V1
Pattern pattern = Pattern.compile(", (\\d{4}), ");
Matcher matcher = pattern.matcher(citation);
matcher.find();
String yearInCitation = matcher.group(1);
writeFullElement(xmlw, dcFlavor+":"+"date", yearInCitation);

... but I need the full YYYY-MM-DD version to put in the the oai_dc output. 🤔 If you have any ideas for me, please let me know.

[pdurbin]

I dug a little more and our citation code is focused on returning just a 4 digit year for the date. This would be a change from what we do now (YYYY-MM-DD)

@philippconzett @plecor @qqmyers what do you think? Should we change oai_dc to YYYY for dc:date?

The spec Philipp found seems to say it's ok. Check out the year 1650 as an example at https://www.base-search.net/about/en/faq_oai.php#dc-date

[qqmyers]

Since the citationDateFieldType is part of the Dataset, I'd think at some point it could/should be part of the DatasetDTO and JSON export, thereby being available to other exporters (will the SPA or other client need this info (in the JSON returned from the dataset api) at some point?).

If that's too much for now, I think the idea of parsing it from the citation as YYYY makes sense, assuming that's sufficient for how people want to use that field. Alternately, I think you could 'go around' the exporter SPI interface and get the full value directly pretty easily as well, e.g. with something like:

        DatasetFieldType citationDataType = jakarta.enterprise.inject.spi.CDI.current().select(DatasetServiceBean.class).get().findByGlobalId(globalId.asString()).getCitationDateDatasetFieldType();
        if(citationDataType!= null) {
            date = dto2Primitive(version, citationDataType.getName());
        } else {
            date = datasetDto.getPublicationDate();
        }

This would not be the only current exporter doing that (e.g. the DDI exporter grabs the ExportInstallationAsDistributorOnlyWhenNotSet Setting it needs).