DM-26680: Schema change integration branch #371
Commits on Sep 26, 2020
-
Only query RUN collections for datasets when asked to query all.
As all datasets are in a RUN collection, this both avoids duplicates and prevents triggering the limitation on CALIBRATION collections in queryDatasets in this common case.
-
Add placeholder queryDatasetAssociations.
This will ultimately need more flexibility in how to accept dataset constraints, rather than just pulling down everything in a dataset type + collection combination, but that is blocked by DM-26692.
-
Re-enable mypy on transfers code and fix breakage.
This was accidentally disabled by moving the code out of core.
-
-
Add isCalibration flag to DatasetType.
This allows old export files to be read, but assumes all dataset types in them have isCalibration=False, which may not be correct.
-
-
Change dataset types in test data.
The "perma" prefixes for these dataset type names made a bit of sense originally because they were attached to "flat" and "bias" dataset types that didn't have a way to be associated with a validity range but otherwise resembled the typical versions of those datasets. But now we're changing how validity ranges are associated with datasets, and there will nothing to distinguish them from the usual datasets, and hence it makes sense to drop that prefix.
-
Remove do-nothing --output_run option from 'butler import'.
The Butler.import_ method never uses the run passed to the Butler at construction, and hence it makes no sense as a command-line argument, let alone a required one. Imported datasets always land in the collections specified in the export file.
-
-
-
Record dynamic table name in dataset type table.
This seems redundant right now, but it provides us some backwards compatibility insulation against changes to how we name those tables or encode dimensions in the future.
-
-
-
-
Change nomenclature for association tables in dataset managers.
There will soon be at least one more kind of dynamic table, so that's no longer an appropriate name for this one.
-
Remove test for exceeding string length
Now that most fields are TEXT the test is meaningless.
-
-
-
Add CALIBRATION-type collections.
General query support for these collections does not yet exist, but is blocked by a bigger overhaul of the query system; we'll work around this in QuantumGraph generation in pipe_base for now.
-
-
Include DatasetType.isCalibration() in __eq__, __repr__.
This helps us spot and report inconsistent definitions in PipelineTasks.
-
-
Support calibration indirection directly in butler.get.
I'm not certain this is the right place for this logic, but the huge simplification it provides for ci_cpp_gen3 tests (which will carry over directly to interactive CPP validation work) suggests that it belongs somewhere in daf_butler.
-
-
Merge pull request #380 from lsst/tickets/DM-26629
DM-26629: switch to calibration collections and remove calibration_label
-
Increment version numbers (again).
This ideally would have been merged with the previous version update prior to the DM-26629 merge, but I forgot, that merge has happened, and so have other DM-26629 merges to master, so it's time to get this branch on master, too.
