API file download issues

[chrowe]

I am trying to download https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OTTBDX# via the API and not having any luck.

I am following the docs at https://guides.dataverse.org/en/latest/api/dataaccess.html

I generated my API token at https://dataverse.harvard.edu/dataverseuser.xhtml?selectTab=apiTokenTab

export API_TOKEN=[my api token]
export SERVER_URL=https://dataverse.harvard.edu
export PERSISTENT_ID=doi:10.7910/DVN/OTTBDX

curl -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID

I have also tried with https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HXZOCG

I just get an empty file named HXZOCG

Running on Ubuntu 16.04.6 LTS

Any ideas why this would not work?

If I just paste the API url into Chrome on my laptop it works fine.

Thanks,
Chris

[poikilotherm]

Some shells ain't happy with the ?. Please try to quote the URL with the vars inside "<url here>" and try again.

[pdurbin]

Hmm, it works fine on the demo site (see example below) so I suspect it's something specific to Harvard Dataverse (both are running Dataverse 5.2). On Harvard Dataverse I tried downloading my tiny "Open Source at Harvard" dataset and it failed in a similar way to the larger ~6 GB dataset @chrowe brought to our attention above (thanks).

Again, here's a working example:

beamish:asdf pdurbin$ export SERVER_URL=https://demo.dataverse.org
beamish:asdf pdurbin$ export PERSISTENT_ID=doi:10.70122/FK2/Q0R60P
beamish:asdf pdurbin$ curl -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4947k    0 4947k    0     0  2863k      0 --:--:--  0:00:01 --:--:-- 2861k
curl: Saved to filename 'dataverse_files.zip'
beamish:asdf pdurbin$ file dataverse_files.zip 
dataverse_files.zip: Zip archive data, at least v2.0 to extract
beamish:asdf pdurbin$ unzip dataverse_files.zip 
Archive:  dataverse_files.zip
   creating: A Logo-02/
  inflating: A Logo-02/A Logo.ai     
  inflating: A Logo-02/A Logo.eps    
  inflating: A Logo-02/A Logo.jpg    
   creating: A Logo-02/ALPHA INTELLIGENCE/
  inflating: A Logo-02/ALPHA INTELLIGENCE/Alpha Intelligence_1.pdf  
  inflating: A Logo-02/ALPHA INTELLIGENCE/Alpha Intelligence_Clean_White_BG.pdf  
  inflating: A Logo-02/ALPHA INTELLIGENCE/Alpha Intelligence_Clean_White_Cropped.pdf  
  inflating: A Logo-02/ALPHA INTELLIGENCE/Alpha Intelligence_Clean_White_Cropped.png  
  inflating: A Logo-02/Read Me.txt   
  inflating: MANIFEST.TXT            
beamish:asdf pdurbin$ 

[djbrooke]

Hey @chrowe - it looks like the answer here is to include -L, like so:

curl -L -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID

On some installations it may work without the -L, but on Harvard Dataverse we have an external zipper service that requires the option.

Would you like to make a PR to update the example in docs? The file in question is here:

https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/api/dataaccess.rst

If not, that's fine, I can create a PR - just let me know. Thanks again for reporting!

[chrowe]

Thanks, this fixed my issue.
The only other change I would suggest is replacing -O with -o dataverse_files.zip. Otherwise the file name is set to last section of the url. e.g. HXZOCG.

So, from my example above the curl command would be

curl -L -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/?persistentId=$PERSISTENT_ID -o dataverse_files.zip

[chrowe]

Well, I may have spoke too soon. The curl command is downloading something that is 2.4GB but when I try to unzip it, I get...

unzip dataverse_files.zip
Archive:  dataverse_files.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of dataverse_files.zip or
        dataverse_files.zip.zip, and cannot find dataverse_files.zip.ZIP, period.

[pdurbin]

@chrowe what do you get if you run file dataverse_files.zip on it?

Also, I thought -J would make the filename right for you. From the man page:

       -J, --remote-header-name
              (HTTP) This option tells the -O, --remote-name option to use the
              server-specified   Content-Disposition   filename   instead   of
              extracting a filename from the URL.

Finally, since this issue close perhaps we should open a fresh one if we end up going back and forth a lot. 😄