DM-24797: Store per-run information (configs, software versions) in butler repo #51
Conversation
PreExecInit now saves task configuratios for every task in a pipeline,
the names of dataset types are fixed as `{taskLabel}_config`. Dataset
types are registered together with all other dataset types.
andy-slac
force-pushed
the
tickets/DM-24797
branch
from
b25ce7a
to
e7c3758
Compare
| Exception | ||
| Raised if ``skipExisting`` is `False` and datasets already | ||
| exists. Content of a butler collection may be changed if | ||
| exception is raised. |
There was a problem hiding this comment.
We could make this exception safe by wrapping it in with self.butler.transaction(), I think.
There was a problem hiding this comment.
Indeed, let me try that.
There was a problem hiding this comment.
That works fine with ci_hsc_gen3, will add that.
| if oldPackages is not None: | ||
| # Note that because we can only detect python modules that have been imported, the stored | ||
| # list of products may be more or less complete than what we have now. What's important is | ||
| # that the products that are in common have the same version. |
There was a problem hiding this comment.
I think the set of imported packages will be sufficiently complete after we've loaded all PipelineTask config instances, and probably not complete before that. Do you happen to know if the sequencing of operations is such that this will be called after that happens?
There was a problem hiding this comment.
I though about that and figured that it should work because the complete graph is in memory (with some caveat) and it means that config instances are loaded too which should import all relevant code.
The caveat is that BPS is executing this not for the whole graph but one task at a time so there is only one task configuration loaded and not whole graph. But this still should work because we update versions incrementally if they don't conflict. And I think BPS should probably do --init-only on the whole graph in the future.
PreExecInit now saves task configurations for every task in a pipeline,
the names of dataset types are fixed as
{taskLabel}_config. It also savespackage versions in a dataset with fixed type name "packages". Dataset
types are registered together with all other dataset types when
--register-dataset-types option is specified.