DM-21939: Create Gen 3 AP Pipeline #55
Conversation
This commit converts the input schema to the catalog-based format needed in Gen 3.
This commit gives the document titles a "(Gen 2)" flag in their title, and renames their files and anchors to avoid conflicts with Gen 3 versions of the same document.
The tutorials referred to old ways of handling data, particularly defects.
This is a direct translation of the pipeline structure, configuration defaults, and configuration validation.
These flags are ignored in Gen 3.
kfindeisen
force-pushed
the
tickets/DM-21939
branch
from
43c50a0
to
812339d
Compare
|
I think I've addressed everything. I also added context paragraphs to the two "getting started" tutorials on cc7e875, if you'd like to take a look. |
| Of course, to inspect this data with the Butler, you don't need to know | ||
| where it lives on disk. You should instead instantiate a Butler within python | ||
| in the ``processed`` directory and access the data products that way. | ||
| contains_no_user_servicable_files/ |
There was a problem hiding this comment.
Ha! Is this really how it works? I couldn't open a calexp or a diffim output by ap_pipe with, e.g., ds9 if I wanted to?
There was a problem hiding this comment.
They're still regular files (for now?), but their organization is more complicated than in Gen 2 and my impression is that Gen 3 is less tolerant of user hacking than Gen 2 was. So I'm disinclined to create the kind of detailed directory tree you had in the original guide (also, as we saw with the collection names, it's very subject to change ATM).
An anchor with this name is likely to appear in another package, such as ``pipe_base``.
|
Thanks for going through all these docs in such detail! All looks good now. |
kfindeisen
force-pushed
the
tickets/DM-21939
branch
from
812339d
to
44244dc
Compare
This PR makes some code changes to conform to lsst/ap_association#86, and adds a pipeline config file to the repository (we may later want to move it to
pipe_tasks, but I think doing so at this point would be confusing). For readability, I've put the documentation in different commits from the changes they document.I've moved the old documentation to "Gen 2" files and created a modified "Gen 3" copy under the original filenames. GitHub displays this as an in-place change; I'm not sure whether it will be easier to review the changes that way, or as a completely new document.