Implement an Embedded Pulsar Job Runner #2057
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pulsar's setup.py has been parameterized to allow it build a special wheel that doesn't depend on galaxy-lib (galaxyproject/pulsar@ce9d4f9) and instead expects it to come from the environment. This special dependency has been published to PyPI here (https://pypi.python.org/pypi/pulsar-galaxy-lib) and added to StarForge here (galaxyproject/starforge#79). These are now available on wheels.galaxyproject.org. In addition to removing the source dirctories - I have removed doc and CI references to these directories.
|
For anyone unclear on why this is cool, this comment from the sample job conf explains it:
In other words, you get all of Pulsar's rewriting and staging functionality directly in Galaxy (it can't stage to remote resources, but it can stage all of a job's necessary files to/from a scratch filesystem). |
|
Awesome! 💯 |
| compute). --> | ||
| <!-- Specify a complete description of the Pulsar app | ||
| to create. Currently this configuration (if set) | ||
| must create exactly on job manager. For more |
There was a problem hiding this comment.
Yup, will open a PR with fix. Thanks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements an initial embedded Pulsar job runner (the last item on #1378 and outlined in this Trello card). See documentation updates to
job_conf.xml.sample_advancedfor information on configuration.In addition to the commit that implements this, PR #2052 is included (since this work builds on that) as well smaller doc and import cleanups for the pulsar job runner code in
lib/galaxy/jobs/runners/pulsar.py.