Figure out how to reasonably appropriate job resources

[afgane]

Per-tool resource request limits are needed. However, it's not a simple table we can compose and use. The current usegalaxy.* efforts use a combination of dynamic rules and tables, which are not straightforward to apply or translate.

Main:

• https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/main/templates/galaxy/config/job_conf.xml.j2
• https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/common/files/galaxy/dynamic_rules/multi_dynamic_walltime.py

.au

• https://github.com/usegalaxy-au/usegalaxy-au-tools/blob/master/tool_destinations.yml

[martenson]

And EU uses

• https://github.com/usegalaxy-eu/sorting-hat (this is what AU uses too)
• https://github.com/usegalaxy-eu/sorting-hat/blob/master/tool_destinations.yaml

To abstract all of this into a common library would be very helpful but challenging task.

[pcm32]

Yes, I have been through this painful process of having to set tool requirements. What I do currently (on LSF) is to have incremental buckets of resources and allow for auto-resubmissions when jobs hit the limit of that resource bucket, automatically sending it to the next bucket. It can be a pain for larger jobs, because they get re-submitted a number of times, but at least I don't any longer need to define specific values per tool.

Disclaimer: I don't think that auto-resubmissions are fully operational with the k8s runner, but since I managed to make it work on the LSF-cli runner, I'm confident that I should be able to get it working soon (as usual, time is scarce).

What happened in the end with the Galactic Radio Telescope? wasn't that suppose to gather usage of resources by tools?

[nuwang]

Integrated with TPV.