Add Docker image as task option #39
Comments
|
Hi @mstone-modulus, thanks for posting this. This shouldn't be too hard to support within redun. The |
|
Hi @mstone-modulus, looking further into this I believe we support task options for |
|
Hi @mattrasmus, Thanks for the quick response, sorry I missed your follow-up. Yes, it looks like you can override the image at the task level, thanks! I do have another related question. If I'm understanding the AWS example docs correctly, each docker image is required to include a
In practice, I'm encountering the following error when running a task inside a Docker container, which is resolved when I install Would it be possible to remove the requirement that each task's image include a Thanks! |
|
Good question. To clarify, @task(executor="batch")
def my_task(x, y, z):
# python code goes here.Script tasks don't need @task
def my_task(x, y, z):
return script(
"my-prog {x} {y} {z}",
executor="batch"
)For a large example see 06_bioinfo_batch, which uses a container without |
|
Thanks again for the quick response. I am actually encountering this error when trying to run a script task. I originally ran into it with a custom image, but here's a reproducible example with a public image. # fastqc.py
from redun import task, script
redun_namespace = 'utils.fastqc'
@task(executor='docker', image='quay.io/biocontainers/fastqc:0.11.9--hdfd78af_1')
def fastqc():
return script("""
fastqc --help
""")And the resulting error log. It looks like the task attempts to check the version of |
|
Thanks for the code example. I believe you can just move the from redun import task, script
redun_namespace = 'utils.fastqc'
@task()
def fastqc():
return script("""
fastqc --help
""",
executor='docker', image='quay.io/biocontainers/fastqc:0.11.9--hdfd78af_1'
)The outer |
|
Thanks! That works. I didn't fully realize there were two tasks here - the decorated function, and the script task it creates - that each have their own executor etc. Makes sense now, thank you for the explanation. One last question on this note - if I'm interested in sometimes running a dockerized task locally (e.g. for testing) and sometimes on Batch, is the most straightforward way to switch between them to change the |
Yes, that's one way to do it switch between @task(executor="batch")
def my_task(...):
...And switch it to: @task(executor="docker")
def my_task(...):
...You could even use env vars as well: EXECUTOR = ("docker" if os.environ.get("TEST") else "batch")
@task(executor=EXECUTOR)
def my_task(...):
...Then you could run: TEST=1 redun run workflow.py main |
|
The last solution seems cleanest to me, thanks for the suggestion. Appreciate all the help! |
Hi,
For dockerized tasks, would it be possible to make the Docker image a task option, rather than an executor option?
e.g.
As far as I can tell, running multiple tasks on Batch in different containers currently requires specifying a separate executor for each task. This feels redundant, since the executor configuration (
queue, etc) is typically the same except for theimage.Thanks!
The text was updated successfully, but these errors were encountered: