AWSBatch Generic Configuration Profile #42
Conversation
conf/awsbatch.config
Outdated
| fixOwnership = true | ||
| runOptions = "-u \$(id -u):\$(id -g)" | ||
| } | ||
|
|
There was a problem hiding this comment.
Docker options are ignored by he AWS Batch executor.
There was a problem hiding this comment.
Thanks @pditommaso ! I adjusted that :-)
There was a problem hiding this comment.
You should also remove docker.enabled :)
main.nf
Outdated
| @@ -58,11 +58,16 @@ def helpMessage() { | |||
|
|
|||
| Other options: | |||
| --outdir The output directory where the results will be saved | |||
| --workDir The temporary directory where intermediate data will be saved | |||
There was a problem hiding this comment.
Nextflow already has a core function for this:
-w, -work-dir
Directory where intermediate result files are stored
There was a problem hiding this comment.
Took care of using that one in 70d4bb0
main.nf
Outdated
| @@ -166,6 +171,11 @@ if( workflow.profile == 'uppmax' || workflow.profile == 'uppmax-modules' || work | |||
| if ( !params.project ) exit 1, "No UPPMAX project ID found! Use --project" | |||
| } | |||
|
|
|||
| //AWSBatch sanity checking | |||
| if(workflow.profile == 'awsbatch'){ | |||
| if (!params.awsqueue || !params.awsregion || !params.workDir.startsWith('s3') || !params.outdir-startsWith('s3')) exit 1, "Specify AWS Batch required parameters!" | |||
There was a problem hiding this comment.
Maybe split this up a bit with different error messages - I could imagine getting super frustrated if I did specify all parameters but used a http address for the outdir or something ;)
Also, minor typo: hyphen instead of a dot: !params.outdir-startsWith('s3')
There was a problem hiding this comment.
Found the typo - already fixed, but I can break it up yes!
|
Really nice stuff 😁 🎉 |
conf/awsbatch.config
Outdated
| } | ||
|
|
||
| batch { | ||
| workDir = params.workDir |
conf/awsbatch.config
Outdated
| */ | ||
|
|
||
| docker { | ||
| enabled = true |
There was a problem hiding this comment.
@pditommaso suggested we remove this 👍
|
Looking great! Now it just needs some documentation 😉 Fine if you make an issue for that and do it separately though. |
|
I added some notes on the command lines at least - will publish a proper blog post on detailed steps to get AWSBatch running (including all steps from zero - running) in the next days with my master student. Plus, I guess we can document the general approach in cookiecutter and also make sure that all pipelines benefit from that documentation? |
|
Will furthermore add a PR to cookiecutter later today... |
This adds a generic AWSBatch configuration profile that allows the users to specify required parameters
Additionally, I implemented a small check next to the UPPMAX sanity checking to check for the existence of these mandatory parameters when running on the "awsbatch" profile, so users get a warning to set these params on AWSBatch properly.
Let me know if that makes sense - happy to polish this further if someone has suggestions!