Deactivate dataset download zip, No error message when reaching :ZipDownloadLimit

[lmaylein]

Is there a way to deactivate the ZIP download for an entire dataset in individual cases?
We have currently set :ZipDownloadLimit to 2GByte and it certainly does not make sense to set a significantly higher value here. But of course we have many datasets with many large files for which it makes no sense to offer an all-in-one-zip-download, especially if it stops when the configured limit is reached.

Wouldn't it be nicer to inform the user when :ZipDownloadLimit is reached, instead of delivering an incomplete zip file without any hint?

How does :ZipDownloadLimit relate to the new zipper tool?

[pdurbin]

@lmaylein yes, what you're describing sounds like a better user experience to me.

When the new zipper tool is in place, an installation can probably increase the :ZipDownloadLimit because the work of zipping is offloaded to the tool (which is outside of Glassfish/Payara). That's been our experience with Harvard Dataverse.

[lmaylein]

@pdurbin
Does that mean :ZipDownloadLimit is also used for the download via zipper tool?

[pdurbin]

@lmaylein no, after looking and the code* and talking to the developer, I can say that when you have the zipper tool in place it pays no attention to the :ZipDownloadLimit setting. Perhaps this is something we could note in the guides where we document that setting. I believe the zipper is still a bit experimental right now but we've been using it at Harvard Dataverse for months.

Circling back to this question:

Wouldn't it be nicer to inform the user when :ZipDownloadLimit is reached, instead of delivering an incomplete zip file without any hint?

I'd be remiss if I didn't point out that this issue is related:

• Download All Error Reporting - Hidden in Manifest if not all files are downloaded (4.9.4) Download All Error Reporting - Hidden in Manifest if not all files are downloaded (4.9.4) #5588

* https://github.com/IQSS/dataverse/blob/v5.3/src/main/java/edu/harvard/iq/dataverse/api/Access.java#L661

[lmaylein]

@pdurbin Unfortunately, we can't get the new zipper to work. The new zip download is invoked, but only creates empty ZIP files. We have therefore switched back to the old download procedure. Is there a newer version than 1.0.0?

[djbrooke]

Hey @lmaylein, can you provide some more details about the failure, and perhaps some logging messages if available?

As @pdurbin mentioned we've been using it for a few months. You may be the first non-Harvard installation to install it, so we're very interested in the experience and any error cases. :)

The version here is the most recent: https://github.com/IQSS/dataverse/tree/v5.3/scripts/zipdownload

[lmaylein]

If we activate the download via the new zipper, we receive a file with the name dataverse_files.zip on every ZIP download.
The size of the ZIP file is only a few bytes and it does not contain any packed files (the ZIP format seems to be fine otherwise).

147.142.xxx.xxx - - [16/Feb/2021:14:15:24 +0100] "GET /cgi-bin/zipdownload?9da-4d1
ac51d1bdd HTTP/1.1" 200 33 "https://heidata.uni-heidelberg.de/dataset.xhtml?pers
istentId=doi:10.11588/data/SXWI6S" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:8
5.0) Gecko/20100101 Firefox/85.0"

Status: 200
Bytes: 33

[qqmyers]

One thing I see in the code (e.g. in DirectAccessUtil) is that it is hardcoded to storage ids of "s3" and "file" - do the installations having problems use different ids?

[lmaylein]

Where can I find the storage id?

This is how it looks in our domain.xml:

        <jvm-options>-Ddataverse.files.file.type=file</jvm-options>
        <jvm-options>-Ddataverse.files.file.label=file</jvm-options>

[qqmyers]

The id is file in your case - it's -Ddataverse.files.<identifier>.type=file
So not the issue for you I guess.

[landreev]

If we activate the download via the new zipper, we receive a file with the name dataverse_files.zip on every ZIP download.
The size of the ZIP file is only a few bytes and it does not contain any packed files (the ZIP format seems to be fine otherwise).

Was /cgi-bin/zipdownload installed on the same host as the Dataverse application?
Even if it is the same server (https://heidata.uni-heidelberg.de), please make sure that the system user running the cgi-bin process (presumably, apache) has read access to the filesystem where the files are stored.

I would also suggest to double-check the database credentials in the zipdownload shell script are correct... But that's probably not an issue, if you were getting an empty zip file. (if the database credentials were bad, you'd be getting either a "no such job" or a database error, and no output, I believe).

So please check that it's not a filesystem access issue. And then, if you are still getting empty files, please check what's in your customzipservicerequest database table, and if it's not empty, send us the last entry or two.

[landreev]

@qqmyers

One thing I see in the code (e.g. in DirectAccessUtil) is that it is hardcoded to storage ids of "s3" and "file" - do the installations having problems use different ids?

It does actually work with multiple stores. For all the stores of types "file" and "s3" the prefixes are converted to "file://" and "s3://" respectively, before they are passed to the zipper. (plus, the full storage path is added to our storageidentifier).
But, as you pointed out, it appears to be a moot point for this installation.

[landreev]

@lmaylein, I do agree with your earlier point - that if the zipdownload limit is being enforced, and the external, cgi-bin zipper is not installed, the "download all" option should be automatically disabled for the larger datasets. Or at least we should give installations an option to disable it.
Would be very easy to implement too of course.

[lmaylein]

please make sure that the system user running the cgi-bin process (presumably, apache) has read access to the filesystem where the files are stored.

That's a good point and might be the problem. I thought the zipper cgi was accessing the files via the API.

[landreev]

... I thought the zipper cgi was accessing the files via the API.

Yes, the zipper gets the files from the filesystem (or S3 bucket) locations directly. The whole point of that "external zipper" experiment was to completely offload that process from the Dataverse application. (We installed it on a separate server, so it reads the files directly off S3, and doesn't compete with the Dataverse application for CPU cycles either).

[landreev]

I also want to mention that we have just merged the PR #7579 into the develop branch, which offers a one-step way to download all the files in a dataset from the command line, that does not use zip at all. (it's an API that presents the files in a dataset as if it were a filesystem directory served by a web server; so you can point wget --recursive at it, and it'll crawl and download all the individual files via standard download API calls, preserving the folder structure etc.)

We don't expect this API to serve as a replacement for zip; since it requires either using wget on the command line, or installing a browser extension that can crawl recursively... Which most users may not be comfortable doing. But for some users it could be a useful alternative.