DM-26680: Schema change integration branch #371
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TallJimbo
force-pushed
the
tickets/DM-26680
branch
4 times, most recently
from
54c8b7c
to
968f832
Compare
TallJimbo
force-pushed
the
tickets/DM-26680
branch
from
968f832
to
b4843bb
Compare
As all datasets are in a RUN collection, this both avoids duplicates and prevents triggering the limitation on CALIBRATION collections in queryDatasets in this common case.
This will ultimately need more flexibility in how to accept dataset constraints, rather than just pulling down everything in a dataset type + collection combination, but that is blocked by DM-26692.
This was accidentally disabled by moving the code out of core.
This allows old export files to be read, but assumes all dataset types in them have isCalibration=False, which may not be correct.
The "perma" prefixes for these dataset type names made a bit of sense originally because they were attached to "flat" and "bias" dataset types that didn't have a way to be associated with a validity range but otherwise resembled the typical versions of those datasets. But now we're changing how validity ranges are associated with datasets, and there will nothing to distinguish them from the usual datasets, and hence it makes sense to drop that prefix.
The Butler.import_ method never uses the run passed to the Butler at construction, and hence it makes no sense as a command-line argument, let alone a required one. Imported datasets always land in the collections specified in the export file.
This seems redundant right now, but it provides us some backwards compatibility insulation against changes to how we name those tables or encode dimensions in the future.
There will soon be at least one more kind of dynamic table, so that's no longer an appropriate name for this one.
Now that most fields are TEXT the test is meaningless.
General query support for these collections does not yet exist, but is blocked by a bigger overhaul of the query system; we'll work around this in QuantumGraph generation in pipe_base for now.
This helps us spot and report inconsistent definitions in PipelineTasks.
I'm not certain this is the right place for this logic, but the huge simplification it provides for ci_cpp_gen3 tests (which will carry over directly to interactive CPP validation work) suggests that it belongs somewhere in daf_butler.
DM-26629: switch to calibration collections and remove calibration_label
This ideally would have been merged with the previous version update prior to the DM-26629 merge, but I forgot, that merge has happened, and so have other DM-26629 merges to master, so it's time to get this branch on master, too.
TallJimbo
force-pushed
the
tickets/DM-26680
branch
from
b574d7b
to
f415ca5
Compare
This was referenced Sep 30, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.