Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-33942: Schema changes #674

Merged
merged 16 commits into from Jun 1, 2022
Merged

DM-33942: Schema changes #674

merged 16 commits into from Jun 1, 2022

Conversation

timj
Copy link
Member

@timj timj commented Apr 7, 2022

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@codecov
Copy link

codecov bot commented Apr 7, 2022

Codecov Report

Merging #674 (754112c) into main (0e52a9b) will decrease coverage by 0.04%.
The diff coverage is 72.72%.

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
- Coverage   84.46%   84.41%   -0.05%     
==========================================
  Files         243      243              
  Lines       31475    31516      +41     
  Branches     5882     5894      +12     
==========================================
+ Hits        26584    26605      +21     
- Misses       3726     3741      +15     
- Partials     1165     1170       +5     
Impacted Files Coverage Δ
python/lsst/daf/butler/registry/tests/_registry.py 99.07% <ø> (ø)
python/lsst/daf/butler/tests/utils.py 94.04% <ø> (ø)
tests/test_cliCmdQueryDataIds.py 95.12% <ø> (ø)
tests/test_cliCmdQueryDatasetTypes.py 96.82% <ø> (ø)
tests/test_cliCmdQueryDatasets.py 95.91% <ø> (ø)
tests/test_cliCmdQueryDimensionRecords.py 96.82% <ø> (ø)
python/lsst/daf/butler/_butler.py 79.78% <31.57%> (-1.57%) ⬇️
python/lsst/daf/butler/transfers/_yaml.py 87.02% <86.66%> (-0.19%) ⬇️
python/lsst/daf/butler/core/ddl.py 83.33% <100.00%> (+0.23%) ⬆️
python/lsst/daf/butler/core/dimensions/_schema.py 90.81% <100.00%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e52a9b...754112c. Read the comment docs.

@timj timj force-pushed the tickets/DM-33942 branch 3 times, most recently from 5678d73 to 4de102a Compare April 11, 2022 19:47
@timj timj force-pushed the tickets/DM-33942 branch 5 times, most recently from d5a2405 to f30a8d7 Compare April 28, 2022 04:08
@timj timj force-pushed the tickets/DM-33942 branch 2 times, most recently from 6207fdb to 4bfe5fe Compare May 5, 2022 22:27
Copy link
Member

@TallJimbo TallJimbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I'm concerned the export/import fixes are necessary-but-not-sufficient.

python/lsst/daf/butler/configs/dimensions.yaml Outdated Show resolved Hide resolved
python/lsst/daf/butler/transfers/_yaml.py Outdated Show resolved Hide resolved
python/lsst/daf/butler/transfers/_yaml.py Show resolved Hide resolved
python/lsst/daf/butler/core/ddl.py Show resolved Hide resolved
@timj timj force-pushed the tickets/DM-33942 branch 5 times, most recently from 1b6ca21 to 5701023 Compare May 27, 2022 21:30
timj and others added 5 commits May 27, 2022 14:33
If a dataset type is a calibration that uses seq_num
it could be satisfied by visit or exposure. In that case
always pick exposure because visit is a derived dimension
and it may not yet have been defined.
A visit can now belong to multiple visit_systems, as long as the set
of exposures computed by those visit_systems is the same.
timj and others added 8 commits May 27, 2022 14:33
It's quite inconveniet to specify a bit mask in a Registry SQL
query so remove it and require that the visit_system_membership
record be used instead.
* Remove visit system from dataset type dimensions.
* Create visit_system_membership records.
A visit that is defined using day_obs/seq_num has an ambiguity
in that it is possible for the one-to-one visit and the
seq_start/seq_end visit to have the same values for day_obs/seq_num.

Now explicitly catch an ambiguous visit and use the default
visit system from the instrument record to determine which one
is likely to be the desired one.
This has to be done for any substantive changes since it is
used in caching.
Having null values in these columns already breaks the query system, so
this should only break unit tests.
This does not catch a later change of primaryKey.
And use that version on import.
Copy link
Member

@TallJimbo TallJimbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, all looks good with the export/import dimensions changes, and I do think keeping the bit of hard-coded translation logic you have (dispatched off the right version, as it now is) is the pragmatic thing to do now, even if it doesn't scale.

@timj timj merged commit 31b4056 into main Jun 1, 2022
@timj timj deleted the tickets/DM-33942 branch June 1, 2022 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants