Skip to content
Dockstore S3 file provisioning plugin
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Dockstore S3 file provisioning plugin


The s3 plugin is capable of download, upload, and can set metadata on uploaded objects.

$ cat test.s3.json 
  "input_file": {
        "class": "File",
        "path": "s3://oicr.temp/"
    "output_file": {
        "class": "File",
        "metadata": "eyJvbmUiOiJ3b24iLCJ0d28iOiJ0d28ifQ==",
        "path": "s3://oicr.temp/"

$ dockstore tool launch --entry  --json test.s3.json
Creating directories for run of Dockstore launcher at: ./datastore//launcher-a246f1b6-21fd-468e-8780-b064d311dda5
Provisioning your input files to your local machine
Downloading: #input_file from s3://oicr.temp/ into directory: /media/large_volume/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-21fd-468e-8780-b064d311dda5/inputs/73b70f11
Found file s3://oicr.temp/ in cache, hard-linking
Calling on plugin io.dockstore.provision.S3Plugin$S3Provision to provision s3://oicr.temp/
Calling out to cwltool to run your tool
Executing: cwltool --enable-dev --non-strict --outdir /media/large_volume/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-21fd-468e-8780-b064d311dda5/outputs/ --tmpdir-prefix /media/large_volu
me/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-21fd-468e-8780-b064d311dda5/tmp/ --tmp-outdir-prefix /media/large_volume/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-
21fd-468e-8780-b064d311dda5/working/ /tmp/1488407859906-0/temp3047430238970788171.cwl /media/large_volume/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-21fd-468e-8780-b064d311dda5/workflow_p
/usr/local/bin/cwltool 1.0.20170217172322
Provisioning your output files to their final destinations
Uploading: #output_file from /media/large_volume/dockstore_tools/dockstore-tool-md5sum/./datastore/launcher-a246f1b6-21fd-468e-8780-b064d311dda5/outputs/md5sum.txt to : s3://oicr.temp/
Calling on plugin io.dockstore.provision.S3Plugin$S3Provision to provision to s3://oicr.temp/
Loading one->won
Loading two->two

Note that metadata is Base64 encoded in the JSON and creates metadata tags on the uploaded file.


This plugin gets configuration information from the following structure in ~/.dockstore/config

endpoint = <endpoint> 

Set the endpoint to a different value in order to talk to a S3 endpoint that is not the official endpoint hosted by AWS. (ex: ) Note that the standard Configuration and Credential Files should be setup in your home directory in order to set an access key and secret access key.

You can’t perform that action at this time.