[16.04] Rev Pulsar to 0.7.0.dev3. #2122
Conversation
jmchilton
commented
Apr 11, 2016
jmchilton
commented
- Fixes numerous problems mostly notably fixes external metadata generation and job metrics collection for recent Pulsar. This was broken when tools were given clean working directory.
- Produce a new message if an older Pulsar version is targeted via REST of library (should help with upgrading to Galaxy 16.04).
- Newer pulsar lib allows more flexible configuration of managers - update documentation to reflect this.
- Fixes numerous problems mostly notably fixes external metadata generation and job metrics collection for recent Pulsar. This was broken when tools were given clean working directory. - Produce a new message if an older Pulsar version is targetted via REST of library (should help with upgrading to Galaxy 16.04). - Newer pulsar lib allows more flexible configuration of managers - update documntation to reflect this.
jmchilton
force-pushed
the
pulsar_metadata
branch
from
3bd50ee
to
84ad9c2
Compare
| @@ -205,6 +206,14 @@ def __init_pulsar_app( self, pulsar_conf_path ): | |||
| log.info("Loading Pulsar app configuration from %s" % pulsar_conf_path) | |||
| with open(pulsar_conf_path, "r") as f: | |||
| conf.update(yaml.load(f) or {}) | |||
| if "job_metrics_config_file" not in conf: | |||
| conf["job_metrics"] = self.app.job_metrics | |||
There was a problem hiding this comment.
job_metrics or job_metrics_config_file ?
There was a problem hiding this comment.
job_metrics is intentional here. If the user doesn't specify an explicit configuration file for Pulsar that is different - I just pass through the actual object (job_metrics) through to Pulsar and it re-uses it - instead of constructing a new one from a configuration file.
There was a problem hiding this comment.
I was wondering because in the if you check for "job_metrics_config_file"
There was a problem hiding this comment.
job_metrics_config_file in conf would mean the deployer specified a file explicitly and so that should be respected. Hence if "job_metrics_config_file" not in conf then a file wasn't specified and the conf object can just be updated to inject Galaxy's job_metrics object directly. I think this correct.
There was a problem hiding this comment.
Ah, ok, that's because Pulsar supports both configuration options (xref https://github.com/galaxyproject/pulsar/blob/master/pulsar/core.py#L129 ). Works for me.
|
@jmchilton If I understand correctly, this does not make it possible to have per-manager or per-destination staging directories, correct? |
|
@natefoo - Umm... still not yet I guess but you should only need one pulsar app/plugin for many destinations/managers now right? Do you still want per-manager staging directories? |
|
Gonna need them to choose between IU and TACC Jetstreams because they have different staging dirs for each. But it's fine, it works as is with two plugins. |
|
I could be missing something - but it is probably as easy as galaxyproject/pulsar#104? With no changes needed on the Galaxy side? Any interest in testing this? I can cut a new version if it looks good. |
|
For anyone reading in the future, this does indeed work with galaxyproject/pulsar#104 and #2184. |
