Auxiliary file download #7729

pdurbin · 2021-03-26T18:17:26Z

What this PR does / why we need it:

Adds the ability to download auxiliary files from the web interface. Also, other cleanup was done.

Which issue(s) this PR closes:

Special notes for your reviewer:

I talked to @djbrooke about trimming down the changes he made but ultimately didn't because I wasn't sure what to take out or how to best word things. I think it's fine to leave it all in but I'm also fine with some of it being removed.

Suggestions on how to test this:

Use AuxiliaryFilesIT or your own methods to add some auxiliary files. Here are the changes to watch for (from the release note):

Auxiliary Files can now be downloaded from the web interface.

Aux files uploaded as type=DP appear under "Differentially Private Statistics" under file level download. The rest appear under "Other Auxiliary Files".

In addition, related changes were made, including the following:

New tooltip over the lock indicating if you have been granted access to a restricted file or not.
When downloading individual files, you will see "Restricted with Access Granted" or just "Restricted" (followed by "Users may not request access to files.") as appropriate.
When downloading individual files, instead of "Download" you should expect to see the file type such as "JPEG Image" or "Original File Format" if the type is unknown.
Downloaded aux files now have a file extension if it can be determined.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Yes. Mockups at https://docs.google.com/presentation/d/1XuX4qStdHjkFf4g1d1vKxxb85kuuUSEzdKN41dNM2mM/

Here's a screenshot showing the DP files:

And another showing "other" files:

Is there a release notes update needed for this change?:

Added.

Additional documentation:

Included.

This is just a stub for now with many FIXMEs.

Not sure why they didn't work before but they work now.

…le option ui clean up [ref #7400, #7469]

…[ref #7400]

All other aux files go under "other".

djbrooke · 2021-03-26T21:03:04Z

Thanks @pdurbin for the PR and for the note about the docs. I took another read through and I think they are fine for now, since we flag this as experimental.

pdurbin · 2021-03-29T14:45:56Z

If the tag for an aux file starts with "dp" it will be displayed under "Differentially Private Statistics" under file level download.

Over at opendp/dpcreator#135 (comment) @TaniaSchlatter asked if this could be changed to "dp_" (adding an underscore) and this got a thumbs up from @raprasad.

Yes, this is certainly possible. Let's see what other changes are requested and do them at once.

pdurbin · 2021-04-06T18:28:39Z

Instead of "starts with dp" we decided to add a proper "type" parameter: d368a85

The downloaded files never had any file extensions. This was fixed in 5c67f95.

Ready for more review.

src/main/webapp/dataset.xhtml

pdurbin · 2021-04-15T20:30:36Z

I made many commits to address the feedback above. In addition I just merged the latest from develop to pick up the change to 5.4.1 and renamed the SQL update script. Ready for more review.

scolapasta · 2021-04-15T20:44:16Z

src/main/java/propertyFiles/Bundle.properties

 restrictedaccess=Restricted Access
+restrictedWithAccessGranted=Restricted with Access Granted


I checked is only uses as a title on an image lock of a file search card; so let's have consistency and just have the one (restrictedaccess=Restricted with Access Granted)

Done in d277aa1. Now the search card is consistent with other places:

scolapasta · 2021-04-16T16:25:29Z

src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java

+    @NamedQuery(name = "AuxiliaryFile.findAuxiliaryFiles",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId"),
+    @NamedQuery(name = "AuxiliaryFile.findAuxiliaryFilesByType",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId and o.type like :type"),


Just noticed this - why "like" and not straight "="

No reason. Fixed in e1d5541.

scolapasta · 2021-04-16T16:29:07Z

src/main/java/propertyFiles/Bundle.properties

+file.downloadBtn.restricted=Restricted
+file.downloadBtn.restricted.with.access.granted=Restricted with Access Granted


We can now get rid of these and just use:
restricted=Restricted
restrictedaccess=Restricted with Access Granted

scolapasta

With the exception of deciding on the API (which we documented as a todo and also explicitly mentioned in the "experimental" section, I think this is good to go. Next up, QA!

pdurbin and others added 17 commits March 9, 2021 12:13

download differentially private statistics #7400

7e7a502

This is just a stub for now with many FIXMEs.

initial docs for DP metadata access

2e89bde

update to match text from depositing data

d8e4b0a

Moved aux file download link to clean up float issue [ref #7400]

53b4201

Moved aux file download link out of canDownload render logic [ref #7400]

5c8085d

Removed aux file download link debug code and typo [ref #7400]

767f2eb

formatTag and formatVersion are working now #7400

3c0140b

Not sure why they didn't work before but they work now.

Render logic changes to file access btn and dropdown options [ref #7400]

0127d5c

Added file access status to dropdown, tooltips to icons, other aux fi…

fc9381a

…le option ui clean up [ref #7400, #7469]

Fixed render logic on Variable Metadata option under File Access btn …

9b7a15d

…[ref #7400]

refactor tests #7400

a04b783

move English to bundle #7400

006a9ba

put aux files starting with tag "dp" under DP Stats #7400

cc00904

All other aux files go under "other".

new domain: opendp.org #7400

5a045be

Add "dp" rule to guides #7400

22f1d2d

add release note #7400

439fcb7

Merge branch 'develop' into 7400-opendp-download #7400

353134f

pdurbin added this to Review 🦁 in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) via automation Mar 26, 2021

pdurbin mentioned this pull request Mar 26, 2021

Distinguishing between "Differentially Private Statistics" and "Other Auxiliary Files" in Dataverse opendp/dpcreator#135

Open

scolapasta moved this from Review 🦁 to IQSS Team - In Progress 💻 in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Mar 30, 2021

scolapasta assigned pdurbin Mar 30, 2021

pdurbin added 6 commits April 1, 2021 15:55

pass isPublic boolean in tests #7400

01741b3

add type for aux files #7400

d368a85

Merge branch 'develop' into 7400-opendp-download #7400

289ef85

add file extension to aux files on download #7400

5c67f95

Merge branch 'develop' into 7400-opendp-download #7400

dcc9104

update release note to reflect recent changes #7400

e419194

scolapasta reviewed Apr 15, 2021

View reviewed changes

src/main/webapp/dataset.xhtml Show resolved Hide resolved

pdurbin added 6 commits April 15, 2021 16:09

remove reference to type=OTHER (legacy concept) in release note #7400

20b4382

reword "type" on aux file page #7400

5dc7234

remove cruft (tmp file that got committed) #7400

51cffe8

remove duplicate query and refactor #7400

b132580

Merge branch 'develop' into 7400-opendp-download #7400

7ea9ffa

rename SQL script (5.4.1 is out) #7400

46d1bd4

pdurbin moved this from IQSS Team - In Progress 💻 to Review 🦁 in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Apr 15, 2021

pdurbin removed their assignment Apr 15, 2021

scolapasta requested changes Apr 15, 2021

View reviewed changes

scolapasta and others added 3 commits April 15, 2021 16:56

minor tweak to wording

20654cd

standardize on "Restricted with Access Granted" #7400

d277aa1

make it clear that API might change #7400

dab2d8b

scolapasta reviewed Apr 16, 2021

View reviewed changes

scolapasta and others added 3 commits April 16, 2021 12:36

Minor text tweaking

ed0b037

switch SQL query from LIKE to = #7400

e1d5541

remove duplicate bundle entries #7400

e913a23

scolapasta approved these changes Apr 16, 2021

View reviewed changes

IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) automation moved this from Review 🦁 to QA 🔎✅ Apr 16, 2021

kcondon self-assigned this Apr 20, 2021

This was referenced Apr 22, 2021

Aux files: Add ability to delete an aux file, esp in cases of correcting mistakes. #7820

Closed

Aux Files: Add some checks to aux files on upload #7819

Closed

add note that aux file APIs can be blocked #7400

cf7b6d1

kcondon merged commit ffc105d into develop Apr 22, 2021

IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) automation moved this from QA 🔎✅ to Done 🚀 Apr 22, 2021

kcondon deleted the 7400-opendp-download branch April 22, 2021 18:28

djbrooke added this to the 5.5 milestone Apr 27, 2021

pdurbin mentioned this pull request May 3, 2021

Auxiliary Files - Accessing DP releases in Dataverse #7400

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auxiliary file download #7729

Auxiliary file download #7729

pdurbin commented Mar 26, 2021 •

edited

djbrooke commented Mar 26, 2021

pdurbin commented Mar 29, 2021

pdurbin commented Apr 6, 2021

pdurbin commented Apr 15, 2021

scolapasta Apr 15, 2021

pdurbin Apr 16, 2021 •

edited

scolapasta Apr 16, 2021

pdurbin Apr 16, 2021

scolapasta Apr 16, 2021

scolapasta left a comment

		restrictedaccess=Restricted Access
		restrictedWithAccessGranted=Restricted with Access Granted

		file.downloadBtn.restricted=Restricted
		file.downloadBtn.restricted.with.access.granted=Restricted with Access Granted

Auxiliary file download #7729

Auxiliary file download #7729

Conversation

pdurbin commented Mar 26, 2021 • edited

djbrooke commented Mar 26, 2021

pdurbin commented Mar 29, 2021

pdurbin commented Apr 6, 2021

pdurbin commented Apr 15, 2021

scolapasta Apr 15, 2021

Choose a reason for hiding this comment

pdurbin Apr 16, 2021 • edited

Choose a reason for hiding this comment

scolapasta Apr 16, 2021

Choose a reason for hiding this comment

pdurbin Apr 16, 2021

Choose a reason for hiding this comment

scolapasta Apr 16, 2021

Choose a reason for hiding this comment

scolapasta left a comment

Choose a reason for hiding this comment

pdurbin commented Mar 26, 2021 •

edited

pdurbin Apr 16, 2021 •

edited