Allow explicit configuration of handlers for workflow scheduling. #3844
Conversation
jmchilton
force-pushed
the
dev_workflow_conf
branch
from
dde6276
to
4edc099
Compare
|
Ping @natefoo 🙏 |
|
👍 to the use of _single_leading_underscore for internal methods. |
lib/galaxy/util/handlers.py
Outdated
| # Parse handlers | ||
| if config_element is not None: | ||
| for handler in self._findall_with_required(config_element, 'handler'): | ||
| id = handler.get('id') |
There was a problem hiding this comment.
I'll put here my usual complain about the reserved word id used as a variable name ;)
There was a problem hiding this comment.
This code was refactored out of jobs/init.py that had the same problem but I'll definitely change it - good catch.
lib/galaxy/jobs/__init__.py
Outdated
| @@ -330,6 +313,12 @@ def __parse_job_conf_xml(self, tree): | |||
|
|
|||
| log.debug('Done loading job configuration') | |||
|
|
|||
| def _parse_handler(self, handler_element): | |||
| for plugin in handler_element.findall('plugin'): | |||
| if id not in self.handler_runner_plugins: | |||
There was a problem hiding this comment.
id (again bad variable name...) is not defined in this method, should be passed as parameter!
There was a problem hiding this comment.
😱 So we would seem to have no test coverage in any of our tests for that. Thanks for the catch 🦅👀!
lib/galaxy/util/handlers.py
Outdated
| def _parse_handler(self, handler_def): | ||
| pass | ||
|
|
||
| def _get_default(self, app, parent, names): |
There was a problem hiding this comment.
You may want to pass config instead of app here, since that's the only part that's used, but not a showstopper.
There was a problem hiding this comment.
I like that change for sure - I'll make it.
I was hoping to follow this up quickly with a switch to YAML - I don't want to write or encourage others to edit more XML. If you insist on documentation, I'm going to rework the configuration files and add it to this PR - but this PR will be less focused as a result. |
Refactor job handler stuff to allow this with re-use.
jmchilton
force-pushed
the
dev_workflow_conf
branch
from
4edc099
to
fd21616
Compare
|
Nice idea! Looking forward to use this. Would be great if you can add some docs in a follow up PR to also use different hosts for wf-scheduling. |
Currently every job handler is a workflow scheduler handler and vice versa. This (in particular bb1bb67) allow specifying a separate (or potentially overlapping) pool of workflow schedulers. Includes lots of testing for figuring out if the current process is a workflow handler and for determining what handler should be assigned for queued workflows.
Builds on the testing from #3820 so it includes that PR and a merge forward of everything into dev.
A long discussion of how this may potentially improve performance, robustness, and quality of logs for workflow scheduling can be found as part of #3816 (comment).