Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTCondor Docker Integration #2278

Merged
merged 7 commits into from May 17, 2016
Merged

HTCondor Docker Integration #2278

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
10ef0a3
HTCondor and Docker documentation
bgruening Apr 30, 2016
ba409e7
enable HTCondor Docker integration
bgruening Apr 30, 2016
3ab43e3
add new Docker Image. We need to have bash inside of Docker.
bgruening Apr 30, 2016
ead96a5
Add the ability to set the default handler and destination via enviro…
bgruening May 1, 2016
7bd595b
Revert "add new Docker Image. We need to have bash inside of Docker."
bgruening May 2, 2016
917c882
Revise HTCondor Docker image resolution.
jmchilton May 16, 2016
8e6211a
Merge pull request #10 from jmchilton/htcondor_docker
bgruening May 16, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 19 additions & 0 deletions config/job_conf.xml.sample_advanced
View file
Open in desktop
Expand Up @@ -434,6 +434,25 @@
<param> tags.
-->
<param id="request_cpus">8</param>

<!-- Recent version of HTCondor do have a `docker` universe to handle containers.
Activate this feature by explicitly specifying the `docker` universe.
-->
<!-- <param id="universe">docker</param> -->

<!-- If the tool has a container specified this one is used.

<requirements>
<container type="docker">bgruening/galaxy-stable</container>
</requirements>

If not the container specified with id="docker_image" is used and as last resort
id="docker_default_container_id" is considered.
-->

<!-- <param id="docker_image">busybox:ubuntu-14.04</param> -->
<!-- <param id="docker_default_container_id">busybox:ubuntu-14.04</param> -->

</destination>

<!-- Jobs that hit the walltime on one destination can be automatically
Expand Down
13 changes: 13 additions & 0 deletions lib/galaxy/jobs/runners/condor.py
View file
Open in desktop
Expand Up @@ -57,6 +57,19 @@ def queue_job( self, job_wrapper ):

# get destination params
query_params = submission_params(prefix="", **job_destination.params)
container = None
universe = query_params.get('universe', False)
if universe.strip().lower() == 'docker':
if job_wrapper.tool.containers:
# Try to extract the container (1) from the Tool, (2) from 'docker_image'
# and (3) from 'docker_default_container_id'. The last two can be specified in job_conf.xml
container = job_wrapper.tool.containers[0].identifier or \
query_params.get('docker_image', False) or \
query_params.get('docker_default_container_id', False)
if container:
# HTCondor needs the image as 'docker_image'
query_params.update({'docker_image': container})

galaxy_slots = query_params.get('request_cpus', None)
if galaxy_slots:
galaxy_slots_statement = 'GALAXY_SLOTS="%s"; export GALAXY_SLOTS_CONFIGURED="1"' % galaxy_slots
Expand Down
4 changes: 2 additions & 2 deletions test/functional/tools/catDocker.xml
View file
Open in desktop
@@ -1,9 +1,9 @@
<tool id="catdc" name="Concatenate datasets (in docker)">
<description>tail-to-head</description>
<requirements>
<container type="docker">busybox:ubuntu-14.04</container>
<container type="docker">bgruening/busybox-bash:0.1</container>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we depended on bash we need a new default container.
The source can be found here: https://github.com/bgruening/docker-busybox-bash

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to move all repos to the Galaxy project.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's revert the bash related changes in lieu of solving the problem of containers not having bash with #2282?

</requirements>
<command>
<command detect_errors="exit_code">
echo "Galaxy slots passed through contain as \$GALAXY_SLOTS";
cat $input1
#for $q in $queries
Expand Down