Refactor base co-execution client out of Kubernetes specific stuff. #302
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmchilton
changed the title
jmchilton
force-pushed
the
more_coexecution
branch
from
df94b84
to
3c2987c
Compare
4 tasks
jmchilton
force-pushed
the
more_coexecution
branch
13 times, most recently
from
c42065a
to
59e0e5a
Compare
Currently the only container-based scheduled (where we schedule containers and not schedule jobs that launch containers) is the Kubernetes message queue based coexecution approach. On one hand, we have some intriguing applications TES and AWS Batch where we would like to schedule containers directly and on the other the MQ approach with Kuberentes can easily be generalized to not require the MQ (falling back to polling the way the Kubernetes job runner in Galaxy does). The goal of this work is to generalize the MQ Kubernetes approach into six approaches: - MQ + Kubernetes (the current recommendation) - Kubernetes w/polling API (a simpler Kubernetes approach that retains all the advantages of the Pulsar approach over the Kubernetes runner in Galaxy without requiring a MQ). - MQ + TES. - TES w/polling. - MQ + AWS Batch - AWS Batch w/polling. TES ---------- I've developed a client library for TES called pydantic-tes (https://github.com/jmchilton/pydantic-tes) - that should use validated models to communicate with a TES server and is tested against Funnel. It also distributes a pytest fixture that can build and launch funnel for writing automated tests and that works with tox and Github actions as demonstrated by the pydantic-tes CI. AWS Batch ---------- TODO: Sequential vs Parallel Container Execution ------------------------------------------- This work contains a generalization of the approach used in Kubernetes of co-execution of Pulsar and Biocontainers, but the model for TES and AWS Batch are more serial container executions - this runs, then that, then that, etc... In TES this is given as a list of "Executors" and AWS Batch has the idea of the job dependencies that I believe can capture this - but this will require a slightly alternative approach (probably simpler) than the K8S co-execution approach in which the containers wait on each other to write files in order to coordinate.
jmchilton
force-pushed
the
more_coexecution
branch
3 times, most recently
from
6cec110
to
c824e30
Compare
…d filter tests. Try to setup TES tests for CI.
jmchilton
force-pushed
the
more_coexecution
branch
from
c824e30
to
ceb1b54
Compare
|
Documentation for this PR: https://pulsar.readthedocs.io/en/latest/containers.html. It includes TES execution. |
jmchilton
added a commit
to jmchilton/galaxy
that referenced
this pull request
Oct 11, 2022
…rnetes. Updating the Pulsar job runner/client for galaxyproject/pulsar#302 and as described in the new docs at https://pulsar.readthedocs.io/en/latest/containers.html.
jmchilton
added a commit
to jmchilton/galaxy
that referenced
this pull request
Oct 11, 2022
…rnetes. Updating the Pulsar job runner/client for galaxyproject/pulsar#302 and as described in the new docs at https://pulsar.readthedocs.io/en/latest/containers.html.
2 tasks
bgruening
changed the title
mvdbeek
pushed a commit
to jmchilton/galaxy
that referenced
this pull request
Oct 16, 2022
…rnetes. Updating the Pulsar job runner/client for galaxyproject/pulsar#302 and as described in the new docs at https://pulsar.readthedocs.io/en/latest/containers.html.
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I want to implement a AWS Batch execution for Galaxy. The existing AWS Batch Runner (galaxyproject/galaxy#13466):
In the reverse direction - adopting this approach to AWS Batch will provide further proof of the viability, encourage me to write documentation, and likely will prove easier to write remote tests for.