DM-49822: Remove Butler from upload_images. #275
Conversation
python/tester/upload_from_repo.py
Outdated
| for visit in visit_list: | ||
| group = increment_group(instrument, group, 1) | ||
| refs = prepare_one_visit(kafka_url, group, butler, instrument, visit) | ||
| uris = [butler.getURI(ref) for ref in refs] |
There was a problem hiding this comment.
In theory butler.get_many_uris is faster than calling getURI in a loop (for direct butler it will do a single database query).
There was a problem hiding this comment.
I wanted to use something like that, but I didn't see that in (daily) pipelines.lsst.io. Did something break the docs?
There was a problem hiding this comment.
There was a problem hiding this comment.
I see, it's in the base class. A shame it doesn't get propagated to the derived.
|
Since this ticket is better practice and works, are we ok to (rebase and then) merge to main? Or is there a way we can wrangle |
|
|
||
| path = transferred[0].path | ||
| path = os.path.join(temp_dir, uri.basename()) | ||
| ResourcePath(path).transfer_from(uri, transfer="copy") |
There was a problem hiding this comment.
I'm not entirely sure what this code is trying to do, but it looks like it's downloading a file from remote to posix path, optionally extracting a sidecar json, and then uploading the raw it to a new location. Is that correct?
Why isn't this doing a direct transfer from one location to another using ResourcePath? Secondly, why isn't the JSON header extracted in the remote using astropy.io.fits with ResourcePath.to_fsspec? Would simplify the code a lot (and use ResourcePath.write without boto calls).
There was a problem hiding this comment.
Not a direct transfer because it needs to modify some headers in the files
There was a problem hiding this comment.
Why are we modifying the headers of raw files?
There was a problem hiding this comment.
Because we are giving them new group ID, new exposure ID, etc to fake new exposures. The script sends a next_visit event with that metadata, and the prompt service will attempt to process the matching exposure (matching via the headers).
kfindeisen
added
the
ignore-for-release
5 participants
No description provided.