DM-43109: Add update-day-obs command #36
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
==========================================
- Coverage 54.35% 52.39% -1.97%
==========================================
Files 33 34 +1
Lines 1148 1212 +64
Branches 255 271 +16
==========================================
+ Hits 624 635 +11
- Misses 490 543 +53
Partials 34 34 ☔ View full report in Codecov by Sentry. |
| visit_defs = butler.registry.queryDimensionRecords( | ||
| "visit_definition", | ||
| where="exposure in (exps)", | ||
| bind={"exps": list(exposures_to_be_updated)}, |
There was a problem hiding this comment.
This potentially needs to be in batches too, but if you are sure that it's not going to be large then it's OK.
There was a problem hiding this comment.
I've been assuming that queryDimensionRecords would be clever enough to do the batching internally if the list is too long, but if you think that's going to be problematic I can chunk it externally.
There was a problem hiding this comment.
My concern here is that WHERE exposure in (...) will become very long, and we do not batch that expression internally.
There was a problem hiding this comment.
I will chunk. I think we do batch internally in some cases and there might even be a ticket for doing it with dimension records.
This implementation is a proof of concept that uses standard butler APIs to do the updates. It may be too slow for large repositories.
Checklist