Download instructions for BALTRAD

[peterdesmet]

Here are download instructions I wrote for BALTRAD. Should be integrated in the website

Data can be downloaded using the following methods.

bioRad

Amazon S3 command line interface

Data are stored in the Amazon S3 bucket enram-vp, which is publicly accessible. The Amazon S3 Command Line Interface (AWS S3 CLI) provides functionalities to efficiently list and download files.

Installation:

python -m pip install awscli

List files for a directory with ls (add --recursive to see the content of subdirectories):

$ aws s3 ls s3://enram-vp/baltrad/hdf5/bejab/
PRE 2018/
PRE 2019/
PRE 2020/
PRE 2021/

Sync/download files to a local directory with sync (add --dryrun to test):

$ aws s3 sync s3://enram-vp/baltrad/hdf5/bejab/ ~/Downloads/baltrad/
download: s3://enram-vp/baltrad/hdf5/bejab/2018/05/22/bejab_vp_20180522T102000Z_0x5.h5 to 2018/05/22/bejab_vp_20180522T102000Z_0x5.h5
download: s3://enram-vp/baltrad/hdf5/bejab/2018/05/22/bejab_vp_20180522T102500Z_0x5.h5 to 2018/05/22/bejab_vp_20180522T102500Z_0x5.h5
download: s3://enram-vp/baltrad/hdf5/bejab/2018/05/22/bejab_vp_20180522T103000Z_0x5.h5 to 2018/05/22/bejab_vp_20180522T103000Z_0x5.h5

Sync is generally more useful than copy (cp), because it will browse subdirectories by default and skip files that are already downloaded. So you can stop the command at any point (Ctrl+c) and it will pick up where it left off if you run it again.