Skip to content

Latest commit

 

History

History
138 lines (88 loc) · 3.23 KB

README.rst

File metadata and controls

138 lines (88 loc) · 3.23 KB

pl-s3retrieve

https://img.shields.io/docker/v/fnndsc/pl-s3retrieve https://img.shields.io/github/license/fnndsc/pl-s3retrieve

A ChRIS ds app to retrieve data of interest from Amazon S3 service

s3retrieve is a ChRIS-based application to retrieve file/folders from Amazon S3 service.

python s3retrieve.py
    [-h] [--help]
    [--json]
    [--man]
    [--meta]
    [--savejson <DIR>]
    [-v <level>] [--verbosity <level>]
    [--version]
    <inputDir>
    <outputDir>
    --bucket <BUCKET>
    --prefix <PREFIX>
    --awskeyid <KEYID>
    --awssecretkey <SECRETKEY>
[-h] [--help]
If specified, show help message and exit.

[--json]
If specified, show json representation of app and exit.

[--man]
If specified, print (this) man page and exit.

[--meta]
If specified, print plugin meta data and exit.

[--savejson <DIR>]
If specified, save json representation file to DIR and exit.

[-v <level>] [--verbosity <level>]
Verbosity level for app. Not used currently.

[--version]
If specified, print version number and exit.

<inputDir>
Input directory.

<outputDir>
Output directory.

--bucket <BUCKET>
Name of the Amazon S3 bucket.

[--prefix <PREFIX>]
If specified, retrieve directory/file prefix path in s3.

--awskeyid <KEYID>
AWS access key id.

--awssecretkey <SECRETKEY>
AWS secret access key.

Getting inline help is:

docker run --rm fnndsc/pl-s3retrieve s3retrieve --man

You need you need to specify input and output directories using the -v flag to docker run.

docker run --rm                                                         \
    -v /tmp/input:/incoming                                             \
    -v /tmp/output:/outgoing                                            \
    fnndsc/pl-s3retrieve                                                \
    s3retrieve --awskeyid KEYID --awssecretkey ACCESSKEY --bucket bch-fnndsc  \
    --prefix test \
    /incoming /outgoing \

The above will retrieve a copy of each file/folder inside the test "folder" in Amazon S3 storage into the local /outgoing directory. Some metadata files should have previously been read from /incoming directory.

Build the Docker container:

docker build -t local/pl-s3retrieve .

Run unit tests:

docker run --rm local/pl-s3retrieve nosetests
https://raw.githubusercontent.com/FNNDSC/cookiecutter-chrisapp/master/doc/assets/badge/light.png