Dataverse: Issues & enhancements to clarify

[joel-simpson]

Problem to be solved.

In the Dataverse project with OCUL we are identifying a number of issues with the current Proof of Concept integration. Many of these need further analysis before we can sort out which are 'defects' and which are enhancement ideas for future projects.
This issue is an umbrella issue for now while we carry out that analysis.

Describe the solution you'd like to see implemented.

From Meghan, a number of data mapping type questions identified so far:

1. Not all of the DDI fields are mapped to METS (missing publication date, abstract/description, keywords, subjects). Note that these are required fields in Dataverse. Can we extend the current mapping?

• TODO: analysis of current data mapping in the wiki

2. A large number of datasets have handles rather than DOIs for the datasets that were deposited prior to version 4.x (see this DV dataset and JSON, for example). Will we be able to capture datasets with handles properly in the METS file under IDNo (pulling persistentURL and protocol from the JSON)?

• I set up an SS space/location for the production version of dataverse, so that I could attempt to retrieve & preserve the QUBS dataset. It fails pretty early on due to a checksum error, but I'm guessing that the prod version may not be up to date with demodv.scholarsportal.info ?

3. The METS contains at least two dmdSec IDs (or more if there are derivatives). For some reason (and this is documented in the PoC) the dmdSecs in the structMap section are not separated between the objects and metadata folders - they are both put on the first line. Is the dmdSec/structMap relationship possible to parse in more detail for subfolders in a transfer?

• TODO: revisit data mapping on wiki

4. For the Bala Parental and Statistical Software datasets, in the objects folder there are zips which contain the original files and DV-generated derivatives, along with metadata files. These zips are also duplicated. See for example, in the StatisticalSoftware AIP, the “MusiciansANOVA.zip” and “MusiciansANOVA.zip-2018-07-19T19_28_21.949093_00_00”. It seems redundant to include both zips in the AIP.

FIXED: this is no longer happening...for any 'bundles', the zips get extracted by the storage service now and only the contents are included in the Transfer to Archivematica.

[ross-spencer]

5. Handle restricted datasets: https://demodv.scholarsportal.info/dataset.xhtml?persistentId=doi:10.5072/FK2/NNTESQ as an example. This cannot be accessed via the API or seen via the public UI. When downloaded, the tree looks like:

115_tab/
├── depress.zip
└── metadata
    ├── agents.json
    └── dataset.json

But the file depress.zip is:

root@artefactual:/tmp/am-pipeline-data/watchedDirectories/activeTransfers/dataverseTransfer/115_tab# file depress.zip 
depress.zip: ASCII text, with no line terminators

And its contents:

root@artefactual:/tmp/am-pipeline-data/watchedDirectories/activeTransfers/dataverseTransfer/115_tab# less depress.zip
Archive:  depress.zip

N.B. The .zip extension is added by the Storage Service.

There are some cues in the metadata we can use:

   "latestVersion":{  
      "files":[  
         {  
            "restricted":true,
            "datasetVersionId":43,
            "version":3,
            "dataFile":{  
               "originalFormatLabel":"SPSS SAV",
               "contentType":"text/tab-separated-values",
               "rootDataFileId":-1,
               "originalFileFormat":"application/x-spss-sav",
               "checksum":{  
                  "type":"MD5",
                  "value":"5fecac4095dbb6374cb32dd70811d2ce"
               },
               "filename":"depress.tab",
               "filesize":5400,
               "UNF":"UNF:6:08UsvQXuLqTKseBaQf77Xw==",
               "md5":"5fecac4095dbb6374cb32dd70811d2ce",
               "id":116,
               "storageIdentifier":"15dc81eb3aa-927d0c6e8e93"
            },
            "label":"depress.tab"
         }
      ],
           
      "termsOfAccess":"Please contact owner for access",

[ross-spencer]

6. Understand why some Dataverse transfers seem to be metadata only in the storage service and what needs to be done to present them better to the user? See below for example storage trees and links to the datasets.

root@artefactual:/tmp/am-pipeline-data/watchedDirectories/activeTransfers/dataverseTransfer# tree 783_empty/
783_empty/
└── metadata
    ├── agents.json
    └── dataset.json

1 directory, 2 files
root@artefactual:/tmp/am-pipeline-data/watchedDirectories/activeTransfers/dataverseTransfer# tree 485_empty/
485_empty/
└── metadata
    ├── agents.json
    └── dataset.json

1 directory, 2 files
root@artefactual:/tmp/am-pipeline-data/watchedDirectories/activeTransfers/dataverseTransfer# tree 25_empty/
25_empty/
└── metadata
    ├── agents.json
    └── dataset.json

1 directory, 2 files

I believe these map to:

• https://demodv.scholarsportal.info/api/datasets/783 (https://demodv.scholarsportal.info/dataset.xhtml?persistentId=doi:10.5072/FK2/ZKDDXA)
• https://demodv.scholarsportal.info/api/datasets/485 (https://demodv.scholarsportal.info/dataset.xhtml?persistentId=doi:10.5072/FK2/NUG7RW)
• https://demodv.scholarsportal.info/api/datasets/25 (https://demodv.scholarsportal.info/dataset.xhtml?persistentId=doi:10.5072/FK2/DEF06J)

[joel-simpson]

7. Enhancement: provide search capabilities from Transfer Browser
   The Transfer Browser is a widget on the Transfer screen where users can select from a list of all Transfer Source Locations, and then browse through directories for the chosen transfer source.
   This design is geared towards file directories. For an object repository like Dataverse, which provides rich search capabilities for users, 'browsing' through datasets may not work well. If there are hundreds or thousands of datasets in a particular dataverse, it is very cumbersome to browse through each.
   Potential enhancement: after selecting a Transfer Type of Dataverse with a Transfer Source Location that is a Dataverse instance, present users with a range of fields that can be used to search the dataverse (these would all have to be parameters avaialble in the Dataverse API) such as author name, dataverse name, subject etc.). Then let users browse those search results (or refine their search further) to find the dataset they want.

[ross-spencer]

8. Double check the expected output for distributor which is currently hard-coded see: Problem: Scholars Portal is hard-coded into Dataverse METS generation #59

[joel-simpson]

9. Verify Transfer Compliance should fail when Transfer type doesn't match transfer Source type
   If a user sets a transfer type of "Standard", but then selects a transfer source that is a dataverse (or more specifically, the transfer source location is associated with a space that has type = dataverse), the Transfer will succeed and continue processing, and only fail at the end of the Ingest stage. This situation should fail at the "Verify Transfer Compliance" stage.

[ross-spencer]

Hi @joel-simpson thinking about 9. to clarify. My expectations as a user would be along these lines:

• Standard Transfer with Dataverse Transfer Source: Pass
• Dataverse Transfer with Standard Transfer Source: Fail

The reason being that a standard transfer should be flexible enough for any data type (there shouldn't be any digital file/group of files that should be able to make it fail)

But the other workflow 'Dataverse' is too specific to be sensible for any group of files not in a Dataverse format.

[gehurley]

Re: issue #2 regarding treatment of handles above

We tested with the production Dataverse with another dataset with a handle - "Research Data Management (RDM) Survey of Queen's University's Engineering and Science Departments." The handle appears to have mapped correctly.

Not sure why the QUBS dataset that Meghan identified in the production Dataverse fails, then, but the handle issue appears to be fine.

[joel-simpson]

10. Multiple authors provided in dataset metadata are not all mapped through to our METS. SP team would like to see this... in line with issue 1 above, this is about ensuring core metadata for search/discovery is available in the METS.

[gehurley]

Re: #1 above regarding extra DDI fields. We'd like to see three additional fields required by Dataverse:

Dataverse	DDI	JSON
Description	abstract	dsDescriptionValue
Subject	subject	subject
Publication Date	distDate	dateOfDeposit

[joel-simpson]

11. Create directories for 'bundles' provided by Dataverse.
    Right now a 'bundle' (.zip with original, derivatives and metadata file for any tabular data file) is simple extracted and included in the first level of a transfer. SP team would prefer to see a directory for each bundle. would make it slightly easier to understand the contents of a transfer and relate to what is displayed in dataverse.