add reworked chronos job runner #11228
Conversation
|
Super! |
|
I created a PR for the docker-compose setup here |
|
Can you fix the linting issues here: |
|
Hi @jmchilton, I've fixed the linting issues. |
|
@mtangaro The kubernetes issue should be fixed if you rebase or merge |
|
Thanks a lot @nsoranzo. Everything is ok now :) |
| <param id="max_retries">2</param> | ||
| directories. Other directories of the data used by tools can | ||
| be mounted as well, separated by commas.--> | ||
| <param id="max_retries">0</param> |
There was a problem hiding this comment.
Any chance I can get you to update the corresponding docs in the sample YAML file (https://github.com/galaxyproject/galaxy/blob/dev/test/unit/jobs/job_conf.sample_advanced.yml). It copied a lot of the comments out of this sample XML but I haven't gotten around to replacing the XML version with the YAML version - but the YAML will be the go to at some point soon.
There was a problem hiding this comment.
This goes without saying! Next time I will update it by default.
|
Thanks so much! |
|
This PR was merged without a "kind/" label, please correct. |
Dear all,
during the last Europe Biohackathon (2020) we reworked the Galaxy Mesos (Chronos) job runner, allowing it to properly work.
Meanwhile, I also did some other improvements and tests.
This is a joint effort by me, @gmauro, @pmandreoli, @maricaantonacci, @aknijn and, of course, @bgruening.
Brief changelog:
Fix the working directory inside the container: the working directory in the container was not right, resulting in the impossibility to retrieve the output files.
GALAXY_SLOTS and GALAXY_MEMORY_MB added to the json and available within the Container.
GALAXY_SLOTS set to docker_cpu
GALAXY_MEMORY_MB set to docker_memory
It has been tested using this documentation page.
A new command script, named chronos_{{job number}}.sh is created instead passing the command in the container preventing wrong characters parsing.
This solve the following issue: Command badly rendered for chronos.py job runner on galaxy 20.05 #10677
Not sure here if this is the right way to implement this, since I'm not using a template and it exploits the $SHELL variable to run it. This variable is correctly handled with biocontainers and, for example, ubuntu images. Any suggestion is more than welcome.
Now there is the possibility to mount multiple path, e.g. galaxy database path and CVMFS reference data, in the Container, with proper permissions.
I added the fail_job function to the runner to correctly show tools stderr in Galaxy. The stderr and stdout are correctly saved in the corresponding file under “outputs”. But, in case of job failure, during the “stop_job” step DB entry with the stderr is overwritten (with nothing).
To fix this, basically, I call the “self._finish_or_resubmit_job” after the “job_state.job_wrapper.fail”, otherwise the last one overwrite the stderr entry in the DB. Please let me know if there is a better way to do this.
Minor fix to job_conf.xml comments and dependencies list.
@gmauro and @maricaantonacci also setup a docker compose here to have it continuously tested. I hope we can do soon a PR for it.
@pmandreoli tested it with the following tools: bowtie2, trimmomatic, trigalore, sort, fastqc, using the galaxyproject CVMFS for the reference data.
To Do list:
this is a first working version, there is still room for improvements. For example, in case of many retries of the same job, the "stop_job" step is not correctly handled.
Marco