As a researcher, I want to enter or upload the provenance for my file so that I can provide other researchers with additional information about the history of my data.

[djbrooke]

The mockups supporting this user story are available here:

https://drive.google.com/file/d/1AO_L0utJwjHEBxpj5b2xNiVkOjOG545D/view

This story covers the submission and validation of a JSON file, and the submission of manually entered provenance.

[pdurbin]

In cc09abf I stubbed out the validation that the prov file is JSON and can be parsed an a JSON object rather than a JSON array.

[dlmurphy]

In our usability testing of our Provenance mockups yesterday, we had one major piece of UI feedback that applied to this mockup:

When you upload a provenance file, the "Upload Provenance File" box is still displayed and active. This led the participant to think that it was possible to upload more than one provenance file. Because he is very familiar with provenance, he was startled to think that we'd messed this up and allowed multiple file upload.

An easy fix is to simply remove the file upload box once a provenance file has been uploaded, and only display it again if the user clicks the "X Remove" button.

[pdurbin]

As of 7948579 we are storing both types of provenance data:

• "Bundle File" (PROV-JSON, a W3C standard): stored as an "auxillary" file for a datafile (like thumbnails)
• "Description" ("free-form", a text box): stored to a new provfreeform column on the filemetadata table

To be clear, this is where the two types will go after you click "Save Changes". There's still staging work to do for the UI. Clicking "Cancel" means nothing that's been staged will be persisted at all.

[dlmurphy]

Usability testing yesterday also got us feedback on some of the instructional text for this popup:

Based on our participant's feedback, I recommend changing this message:

"Please add any information required to accurately document the history of this data file, including how it was created and how it was transformed."

TO

"In addition to a bundle file, or in place of one, you may add any information required to accurately document the history of your data file, including how it was created and how it was transformed."

I also recommend changing Description to Provenance Description.

[dlmurphy]

Lastly, @mheppler pointed out during usability testing that we need a "Cancel" button added to this popup modal:

[mheppler]

What we are building

Include recommendations from testing as outlined above by @dlmurphy and ok'd by @TaniaSchlatter and @djbrooke and the following:

File Upload Pg

• Edit button for each file with dropdown menu for Provenance (combine "Edit Tags" button as a dropdown option, as well as Delete, Restrict and Unrestrict)
• New Edit dropdown button in the table header, consistent with the Edit File dropdown button in the files table header on the dataset pg (the Provenance option will not be linked from this button, we are not supporting batch/multi-file editing for Provenance)

Provenance (Edit) Popup

• New Provenance popup will provide help text and form input for the "Provenance Bundle" and the "Provenance Description", along with Save Changes and Cancel buttons
• Provenance Bundle form input is a file upload widget that validates JSON format (a nice to have is to validate against the W3C schema)
  • Upon uploading valid bundle, the upload widget is hidden, and a Remove and Preview button are displayed next to the bundle file name
  • Remove button removes uploaded bundle file, the upload widget is then displayed again
  • Preview button opens Provenance graph in new window
• Provenance Description form input is a textarea component
• Save Changes stages changes as part of the File Upload workflow
• Cancel deletes changes

Provenance (Confirmation) Popup

• Warning text explaining user must preview the Provenance info they added before publish, because you can not change it after you publish your dataset
• Continue button brings user back to File Upload workflow
• Cancel button brings user back to Provenance (Edit) popup

[TaniaSchlatter]

When we have a better sense of the graph, it may make more sense and streamline the flow to include the Preview in a modal.

[mheppler]

Mockup of the Edit dropdown button in the table header for File Upload pg, as referenced in checklist above.

This is intended to bring the functionality of the same Edit Files dropdown button on the dataset pg to the File Upload pg.

[matthew-a-dunlap]

Sorry, meant to delete a comment not close the story!

[mheppler]

Thank you @kcondon and @matthew-a-dunlap for coordinating on the Chrome browser issues and resolving those as merely browser CSS caching issues.

Found a minor UI issue and demo'd it for Matthew on dataverse-internal...

• If prov file previously saved for unpublished draft, when you go back to prov popup the "Data File Entity" selection is not retained in the dropdown.

[matthew-a-dunlap]

Here are the issues I've fixed but should be QA'ed again:

• Prov desc does not show up on file versions tab...
• Server log error on edit popup if access via edit file metadata...

[kcondon]

OK, tested the above fixes and they work.

[matthew-a-dunlap]

More fixes info:

• Fixed: "If prov file previously saved for unpublished draft" (Mike's)
• I think I fixed but couldn't quite reproduce: "While editing prov from edit file metadata page, if add prov file"
• Fixed: All API issues. As part of this I removed the DeleteProvFreeform API command as it was redundant/unneeded (update can be used for the same functionality). Also note that Prov JSON can be uploaded for a published DataFile as long as JSON doesn't already exist.

I have addressed all the issues I am aware of at this point. We'll see what else comes up!

[kcondon]

All of the above are fixed. A minor note about maybe providing a curl example for create prov, showing --upload-file

[matthew-a-dunlap]

After discussion with @scolapasta and @sekmiller , it looks like one of the saving flows on the editDatasetPage is not saving with best practice and the fileMetadata is likely not in a useful state afterwards for the prov command to use and save. Its possible prov could work around this but it'd be pretty messy, I'm going to first take a look at what a SaveDataFileCommand would look like.

[matthew-a-dunlap]

After talking with @landreev , I was able to arrive at a fix that did not involve digging deep into saving commands. Prov changes now trigger the full dataset update in editDatafilePage, instead of the optimized but less robust flow.

When investigating this change, here are the things to look for:

• Saving prov json and prov freeform from editDatafilePage after a file upload (not editing prov during the upload action) should work.
  • Try doing this with a tabular file and tabular data tags, there was some change these could conflict with each other tho from my testing it looked ok
• Along with this I found a bug where editing just the entityId did not save. Should work now.
• Please reconfirm other prov popup flows, especially ones on the editDatafilePage. I ran through them but I often seem to miss things.

I was unable to reproduce the windows prov file issues in windows 8 on IE and Chrome (Edge is not available for windows 8). While investigating I uncovered an issue on the file page where repeatedly saving the prov popup to delete and then create prov json didn't not clean up completely and caused entity saving to fail.

[kcondon]

Tested the above cases and all work. Mike confirmed his issue is fixed. Win browser issues appear to be caching related. Now retesting other workflows, then merge!