DM-33937: Setup a butler with real data on google cloud #13
Conversation
|
|
||
| args = _make_parser().parse_args() | ||
| seed_config = lsst.daf.butler.Config(args.seed_config) | ||
| logging.info("Creating repository at %s...", args.target_repo) |
There was a problem hiding this comment.
Why does the make_local_butler.py file exist? Isn't that identical to this version if the default value for --target-repo was "."? Have you considered making this command a butler pluggable command so that you could make use of the normal butler --help click infrastructure? (it's butler create + butler import).
There was a problem hiding this comment.
I do not believe that integrating what should be a local, highly specific script into the butler ecosystem would be beneficial, even if we had the time to do so.
We are deliberately not using the butler command-line itself (i.e., this is not a shell script) because we need to understand what our overheads are, and the butler utility loses a lot of time on package import and Butler setup.
| """ | ||
| butler = daf_butler.Butler(repo) | ||
| with butler.export(format="yaml") as contents: | ||
| # Need all detectors, even those without data, for visit definition |
There was a problem hiding this comment.
I'm not sure I understand this comment. I'd think we could pretend for the purposes of any given repo that a camera has a subset of its actual detectors.
There was a problem hiding this comment.
This is something carried over from ap_verify. I don't know if it is still true, but visit definition used to fail if any of the detector IDs in the instrument (camera?) definition were missing from the registry.
The file can be used to create a reproducible dump of any ap_verify dataset's preloaded repo.
kfindeisen
force-pushed
the
tickets/DM-33937
branch
3 times, most recently
from
cff1146 to
8afbfaa
Compare
This will allow HSC raws to be generated to put into the central repo.
The Playbook is the only place where we can practically document the buckets at present.
kfindeisen
force-pushed
the
tickets/DM-33937
branch
from
8afbfaa to
5424fb8
Compare
4 participants
This PR provides scripts for distilling
ap_verify_ci_hits2015into only the data we want to put into the Prompt Processing "initial" repo. Note thatmake_remote_butler.pycan only be run on Google Cloud.