Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-32061: Fix some issues with export-calibs command #703

Merged
merged 9 commits into from Jun 29, 2022
Merged

DM-32061: Fix some issues with export-calibs command #703

merged 9 commits into from Jun 29, 2022

Conversation

timj
Copy link
Member

@timj timj commented Jun 28, 2022 •
edited

  • Complain if a template uses record metadata but no record metadata available.
  • Add --transfer and --datasets to butler export-calibs
  • Add collection defaulting.
  • Expand dataset refs in export-calibs

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@codecov
Copy link

codecov bot commented Jun 28, 2022 •
edited

Codecov Report

Merging #703 (c893ea6) into main (3ec7b18) will decrease coverage by 0.00%.
The diff coverage is 60.00%.

@@            Coverage Diff             @@
##             main     #703      +/-   ##
==========================================
- Coverage   84.54%   84.54%   -0.01%     
==========================================
  Files         243      243              
  Lines       31785    31814      +29     
  Branches     5954     5963       +9     
==========================================
+ Hits        26874    26896      +22     
- Misses       3740     3747       +7     
  Partials     1171     1171
Impacted Files Coverage Δ
python/lsst/daf/butler/datastores/fileDatastore.py 80.93% <0.00%> (-0.45%) ⬇️
python/lsst/daf/butler/script/exportCalibs.py 14.54% <10.00%> (-1.79%) ⬇️
python/lsst/daf/butler/cli/cmd/commands.py 91.33% <100.00%> (+0.05%) ⬆️
python/lsst/daf/butler/core/fileTemplates.py 78.17% <100.00%> (+1.46%) ⬆️
tests/test_butler.py 97.60% <100.00%> (+0.01%) ⬆️
tests/test_datastore.py 99.48% <100.00%> (ø)
tests/test_templates.py 98.84% <100.00%> (+0.03%) ⬆️
python/lsst/daf/butler/core/dimensions/_records.py 85.62% <0.00%> (+0.65%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ec7b18...c893ea6. Read the comment docs.

@timj
Add metric5 for testing
9caef84 
This is needed in standalone datastore tests because the metric2 template
includes visit.name and standalone datastores do not have expanded
dataset refs.
@timj timj marked this pull request as ready for review June 29, 2022 16:18
@timj timj requested a review from czwa June 29, 2022 16:18
czwa
czwa approved these changes Jun 29, 2022
View reviewed changes
Copy link
Contributor

@czwa czwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took a bit for me to understand the butler features I don't regularly run into, but the export code looks reasonable. I think there's one missing docstring entry (noted in its own comment).

python/lsst/daf/butler/core/fileTemplates.py Show resolved Hide resolved
timj added 8 commits June 29, 2022 14:31
@timj
Complain if a file template uses a dimension record but that record i…
c45e97c 
…s missing
@timj
Expand dataIds when exporting
c2420c7 
In some scenarios a file template will be used to create the export
file name and that will need metadata attached to records in our
default templates.
@timj
Update the DIRECTORY help text to indicate output
9cd3005
@timj
Add --dataset option to export-calibs
63d8a64 
This lets you restrict the dataset types to be exported.
@timj
Default collections to all CHAIN/CALIBRATION collections if not speci…
89d58b8 
…fied
@timj
Add transfer option to export-calibs
7a05714
@timj
Disable move export mode and assume direct == None for export
c6af64a 
move should never be allowed for export since the files must
still stay with the original datastore.

direct is also dangerous since we do not want files that
could be moved out of the datastore on import and we do not
want a direct import to have files disappear if the original
butler no longer wants the file.
@timj
Add news fragment
c893ea6
@timj timj merged commit aeba0b4 into main Jun 29, 2022
@timj timj deleted the tickets/DM-32061 branch June 29, 2022 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@czwa czwa czwa approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@timj @czwa