Harvesting - Broken dataset title links for non-Dataverse/OAI-PMH repositories

[jggautier]

When Dataverse harvests from some non-Dataverse sources (e.g. ICPSR, DataCite), clicking on the dataset link doesn't take users to the source's dataset page.

You can see an example in this dataverse on Harvard Dataverse, where a set of metadata records from DataCite was harvested. Clicking on the dataset title link (https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.6141/tw-srda-af010014-1) take you to DataCite's 404 Not Found page (https://oai.datacite.org/dataset.xhtml?persistentId=doi:10.6141/tw-srda-af010014-1). Clicking on the DOI link (https://doi.org/10.6141/tw-srda-af010014-1) in the citation box takes you to the source's dataset page.

(On Harvard Dataverse's "ICPSR Harvested Dataverse", clicking on the dataset titles correctly takes you to the sources' dataset pages. I'm not sure how those dataset records were collected into that ICPSR Harvested Dataverse. But they're out of sync with the records that ICPSR makes available over OAI-PMH, and this bug is prevent Harvard Dataverse from updating the datasets it's harvested from ICPSR.)

Could we investigate what's going on?

These are two github issues that might be related:
#4831
#4707

[mheppler]

Stumbled upon this issue again today. We are up to 2,470 datasets that result in 404's when you click on the dataset title link in the search card.

The resulting 404 URL is formatted with the assumption that there is a "dataset.xhtml" page on the DataCite site.

https://oai.datacite.org/dataset.xhtml?persistentId=doi:10.6141/TW-SRDA-E89101-1

Looking at the code, it appears there is return remoteArchiveUrl; logic but it appears that it isn't being properly applied.

search-include-fragment.xhtml

<!--DATASET CARDS-->
                    <div class="datasetResult clearfix" jsf:rendered="#{result.type == 'datasets'}">
                        <div class="card-title-icon-block">
                            ...
                            <a href="#{!SearchIncludeFragment.rootDv and !result.isInTree ? result.datasetUrl : widgetWrapper.wrapURL(result.datasetUrl)}" target="#{(!SearchIncludeFragment.rootDv and !result.isInTree and widgetWrapper.widgetView) or result.harvested ? '_blank' : ''}">
                                <h:outputText value="#{result.title}" style="padding:4px 0;" rendered="#{result.titleHighlightSnippet == null}"/>
                                ...

SolrSearchResult.java

    public String getDatasetUrl() {
        String failSafeUrl = "/dataset.xhtml?id=" + entityId + "&versionId=" + datasetVersionId;
        if (identifier != null) {
            /**
             * Unfortunately, colons in the globalId (doi:10...) are converted
             * to %3A (doi%3A10...). To prevent this we switched many JSF tags
             * to a plain "a" tag with an href as suggested at
             * http://stackoverflow.com/questions/24733959/houtputlink-value-escaped
             */
            String badString = "null";
            if (!identifier.contains(badString)) {
                if (entity != null && entity instanceof Dataset) {
                    if (this.isHarvested() && ((Dataset)entity).getHarvestedFrom() != null) {
                        String remoteArchiveUrl = ((Dataset) entity).getRemoteArchiveURL();
                        if (remoteArchiveUrl != null) {
                            return remoteArchiveUrl;
                        }
                        return null;
                    }
                }
                if (isDraftState()) {
                    return "/dataset.xhtml?persistentId=" + identifier + "&version=DRAFT";
                }
                return "/dataset.xhtml?persistentId=" + identifier;
            } else {
                logger.info("Dataset identifier/globalId contains \"" + badString + "\" perhaps due to https://github.com/IQSS/dataverse/issues/1147 . Fix data in database and reindex. Returning failsafe URL: " + failSafeUrl);
                return failSafeUrl;
            }
        } else {
            logger.info("Dataset identifier/globalId was null. Returning failsafe URL: " + failSafeUrl);
            return failSafeUrl;
        }
    }

[djbrooke]

• The URLs assume a dataset link for each harvested record which is probably the issue
• This should be addressed through a configuration change (not code change hopefully) of the harvesting client and verify
• We should identify what clients we expect to change in Harvard Dataverse
• We should add/verify docs

[pdurbin]

@landreev at standup I mentioned that I was chatting with @jggautier about this issue this morning.

One thing we noticed is that when I tried to set up a harvesting client from https://oai.datacite.org/oai it was taking FOREVER after I clicked "Next".

Actually, the same thing happens when I click "Next" under "Edit Harvesting Client" like this. It just spins and spins:

But then Julian clued me in to the fact that https://oai.datacite.org/oai has over 2000 sets. In order to know how long I should wait (10 minutes or so?) I hacked in a counter like this:

$ git diff
diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java
index e4642fe0a..bd805bef2 100644
--- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java
+++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java
@@ -157,7 +157,10 @@ public class OaiHandler implements Serializable {
         
         List<String> sets = new ArrayList<>();
 
+        int count = 0;
         while ( setIter.hasNext()) {
+            count++;
+            System.out.println("on set " + count);
             Set set = setIter.next();
             String setSpec = set.getSpec();
             /*

Obviously, the code above is a hack but I guess I'd suggest adding in some more logging (logger.fine, probably) if you feel like it, while you're in this code.

Also, I can easily reproduce the bug with the client above. Here are all the parameters I used:

• harvestingurl: https://oai.datacite.org/oai
• harvestingset: GESIS.SRDA
• harvesttype: oai
• metadataprefix: oai_dc
• harveststyle: dataverse

[jggautier]

This should be addressed through a configuration change (not code change hopefully) of the harvesting client and verify

This seems to be the case. I'm helping resolve a different harvesting issue and I noticed that when setting up a harvesting client, the last of 4 steps is to choose the "Archive Type." When I set up harvesting of non-Dataverse repositories, I left the Archive Type as Dataverse v4+, and it seems like I was expected to choose "Generic OAI resource (DC)".

ICPSR datasets are in the middle of being harvesting into this dataverse on Demo Dataverse, and the dataset title links I've clicked take me to the records on ICPSR's website. (I chose "Generic OAI resource (DC)" since I didn't know why there was an option specifically for ICPSR.)

I think that needing to choose "Generic OAI resource (DC)", in step 4, also implies that in step 2, I was expected to choose oai_dc as the metadata format.

We should identify what clients we expect to change in Harvard Dataverse

The non-Dataverse repositories that Harvard Dataverse should re-harvest are:

• SRDA (which is a set in DataCite's oai-pmh feed, and Harvard Dataverse might be having a problem getting the large list of sets in DataCite's feed, as @pdurbin wrote earlier)
• ICPSR (since the existing records in Harvard Dataverse are stale, and hopefully using OAI-PMH can keep them up to date)

We should add/verify docs

The docs (http://guides.dataverse.org/en/latest/admin/harvestclients.html) don't mention each step. I'm not sure if there's a need for them to.

In that screenshot, the modal window describes the importance of choosing the right Archive Type (I think I just overlooked it since I don't normally set up harvesting from non-Dataverse repositories):

Maybe not making Dataverse v4.x as a default will force the user to think about the Archive Type.

[mheppler]

@jggautier, kudos for the comment with screenshots, proposing UX/UI improvements to the create harvesting client workflow in order to avoid this issue going forward.

It should be easy enough to change the dropdown menu in Step 4 to be "Select...", forcing the user to make a selection. Are you also suggesting that we could combine Step 2 and Step 4 because of the relation of Metadata Format and Archive Type fields?