Can Caper upload data for cloud backend?

[yihming]

Hi,

I'm just wondering if Caper has the functionality of uploading data when submitting a job to cloud. Or users need to upload using cloud SDK themselves, then submit jobs by referencing file locations on the cloud bucket? Thanks!

Sincerely,
Yiming

[leepc12]

Caper automatically localizes (making copies on the destination cloud storage for files defined in the input JSON) if the cloud platform is one of the followings: gs://, s3:// or URLs. Make sure to choose a correct backend in Caper's conf (~/.caper/default.conf) and authenticate yourself with platform CLIs (gcloud auth or aws configure, ...).

[yihming]

If I specify local data files in the input JSON, and use GCP or AWS backend, will caper upload local files to cloud first?

[leepc12]

Yes, make sure that you define parameters correctly for the specific backend.

For example, if the following parameters are defined in your conf file then Caper will localize your local data files (say, /data/rep1.fastq.gz) on gs://your-output-bucket/output-folder/.caper_tmp keeping the original directory structure (gs://your-output-bucket/output-folder/.caper_tmp/data/rep1.fastq.gz) and make a temporary input JSON file to be submitted to the Cromwell engine. Make sure to authenticate (with cloud's CLIs. e.g. gcloud auth, aws configure for private s3://, ~/.netrc for private URLs) your account on the system where you run caper command lines.

backend=gcp
gcp-out-dir=gs://your-output-bucket/output-folder

[yihming]

It's great to know that Caper does have this feature. Your information is very helpful. Thank you very much!