Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job runner for Chronos #3946

Merged
merged 4 commits into from May 18, 2017

Conversation

Projects
None yet
6 participants
@theosotr
Copy link
Contributor

commented Apr 20, 2017

Chronos (https://mesos.github.io/chronos/) is a framework for the Apache Mesos (http://mesos.apache.org/); a software which manages computer clusters. Specifically,
Chronos is a batch processing scheduler that runs on top of Mesos and it is used for job
orchestration.

Add job runner for Chronos
Chronos is a framework for the Apache Mesos; a software which manages
computer clusters. Specifically, Chronos is a batch processing
scheduler that runs on top of Mesos and it is used for job
orchestration.

@galaxybot galaxybot added the triage label Apr 20, 2017

@galaxybot galaxybot added this to the 17.05 milestone Apr 20, 2017

<param id="volumes">/directory/</param>
<!-- Directory which is mounted to the container and is parent of
the `job_working_directory`, `file_path`, `new_file_path`
directories -->

This comment has been minimized.

Copy link
@jmchilton

jmchilton Apr 20, 2017

Member

Can you update this comment to mention reference data as well - everything mentioned in .loc files (if they are used by these jobs) need to be included as well.

@jmchilton

This comment has been minimized.

Copy link
Member

commented Apr 20, 2017

This is utterly fantastic - while I think that we could open up the deployment opportunities by integrating this with Pulsar instead since we could map everything into that volumes directory as needed - I really do appreciate you very clearly lay out the requirements to use this job runner and avoided hacks to do file staging (which Galaxy makes difficult).

Once we get that client library in starforge - I'll be a +1 on this. I don't think we have how to do that documented anywhere though - so I can try to find some time to do it.

@bgruening bgruening referenced this pull request Apr 20, 2017

Merged

Add chronos wheel #126

@jmchilton

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

@martenson martenson modified the milestones: 17.09, 17.05 Apr 26, 2017

@@ -69,3 +69,6 @@ ecdsa==0.13

# Flexible BAM index naming
pysam==0.8.4+gx5

# Chronos client
chronos-python

This comment has been minimized.

Copy link
@jmchilton

jmchilton May 1, 2017

Member

Will you pin the version (use the line chronos-python==0.38.0 instead) please?

@bgruening

This comment has been minimized.

Copy link
Member

commented May 17, 2017

@jmchilton was this a +1 from you?
For me this looks great, it's documented and isolated enough to get it in.

@jmchilton jmchilton merged commit 7028ca0 into galaxyproject:dev May 18, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 19, 2017

chronos-python has a dependency on httplib2>=0.9, which is not available in https://wheels.galaxyproject.org/ and is not listed in our requirements files.

Also, these 2 requirements can maybe go under lib/galaxy/dependencies/conditional-requirements.txt instead of pinned-requirements.txt, with the appropriate code added to lib/galaxy/dependencies/__init__.py, what do you think?

@jmchilton

This comment has been minimized.

Copy link
Member

commented May 19, 2017

@nsoranzo - yup - these need to be done. I've created an issue for this #4085.

@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 19, 2017

Thanks @jmchilton!

@jmchilton

This comment has been minimized.

Copy link
Member

commented May 19, 2017

@nsoranzo Out of curiosity how did you notice? Is there some automated check we could quickly implement to ask if Galaxy's dependencies are all available as wheels Galaxy-hosted wheels?

@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 19, 2017

I noticed my dev Galaxy downloaded 2 new wheels after a git pull, I remembered this PR but not about httplib2, went to check on https://wheels.galaxyproject.org/ and didn't find it, so manual 🦅 👁 process ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.