DM-30624: Document gen2 to gen3 refcat ingestion #260

leeskelvin · 2021-10-14T16:21:26Z

No description provided.

parejkoj · 2021-10-18T21:16:06Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+        import glob
+        import astropy.table
+
+        refcat_dirs = [gaiadr2, panstarrsps1]


These need to be strings, and are not the names we have for these on lsst-devl (see /datasets/refcats/htm/v1).

They're meant to be objects, that the user has already assigned elsewhere. I hoped the intro text to this code block explained that. As this code is intended to be used on other machines, where the refcats may be saved in a non-standard location, I thought this was the safest bet.

If it would help, I could add some dummy lines above in the script along the lines of:

gaiadr2 = "/path/to/my/gaia/dir" panstarrsps2 = "/path/to/my/panstarrs/dir"

parejkoj

Several comments. In particular, I think we want this as a separate section, as the note really breaks the text flow.

Have you run the script to test it?

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

parejkoj

Please modify the first note in the document (just before section 1) to point to the new section.

Several more rephrasing suggestions.

parejkoj · 2021-10-20T19:46:24Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst


 .. _lsst.meas.algorithms-refcat-ingest:

 4. Ingest the files into the butler
 ===================================

-When ``convertReferenceCatalog`` has finished, it will print the two commands you need to run to register the new refcat dataset type and ingest your converted output into it.
+When ``convertReferenceCatalog`` has finished, a new directory (named ``gaia-refcat/`` in the example above) will now exist containing all reference catalogs in the LSST format.


"... containing the HTM-indexed files for the input catalog in the LSST format."

parejkoj · 2021-10-20T19:47:56Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+
+.. _lsst.meas.algorithms-refcat-existing:
+
+5. What to do with existing reference catalogs?


I know I suggested it, but maybe: "Ingesting pre-existing gen2 reference catalogs"

parejkoj · 2021-10-20T20:28:29Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+5. What to do with existing reference catalogs?
+===============================================
+
+Already existing reference catalogs (for example, the PS1 or Gaia DR2 catalogs that were used in gen2 butlers) can also be ingested into the butler, negating the need to convert them to the LSST format yourself.


Maybe "... can be directly ingested into a gen3 repo as they are already in the LSST format."

parejkoj · 2021-10-20T20:32:43Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+===============================================
+
+Already existing reference catalogs (for example, the PS1 or Gaia DR2 catalogs that were used in gen2 butlers) can also be ingested into the butler, negating the need to convert them to the LSST format yourself.
+To ingest an already existing converted reference catalog, first create a suitable filename to htm7-index astropy lookup table, and then follow the steps above to `ingest the files into the butler <lsst.meas.algorithms-refcat-ingest>`.


The link at the end of this sentence doesn't work in the generated output. I think you need a :ref: in front.

parejkoj · 2021-10-20T20:34:33Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+Already existing reference catalogs (for example, the PS1 or Gaia DR2 catalogs that were used in gen2 butlers) can also be ingested into the butler, negating the need to convert them to the LSST format yourself.
+To ingest an already existing converted reference catalog, first create a suitable filename to htm7-index astropy lookup table, and then follow the steps above to `ingest the files into the butler <lsst.meas.algorithms-refcat-ingest>`.
+
+This is an example script that constructs a conversion lookup table for all converted FITS files in the `refcat_dir` directory, outputting to the `out_dir` directory:


"that constructs..." -> that creates an ``.ecsv`` file for the ``butler ingest-files`` command, from all of the HTM indexed files in a given directory. We use the existing Gaia DR2 catalog on lsst-devl in this example:

This is an example script that creates an .ecsv lookup table for the butler ingest-files command, from all of the HTM indexed files in a given directory (refcat_dir here). We use the existing Gaia DR2 catalog on lsst-devl in this example:

parejkoj · 2021-10-20T20:35:33Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+
+    table.write(out_file)
+
+Once this script is complete, finalize the reference catalog ingestion by following the `file ingestion instructions above <lsst.meas.algorithms-refcat-ingest>`.


"finalize"->"finish", maybe?

And the final link doesn't work in the output HTML. I think you need a :ref: before these.

https://developer.lsst.io/restructuredtext/style.html#linking

parejkoj · 2021-10-20T20:37:09Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+    import glob
+    import astropy.table
+
+    refcat_dir = "/path/to/my/refcat/directory"


In my comment above, I suggested using /datasets/refcats/htm/v1/gaia_dr2_20200414 here, to make it clear what exactly the this path should look like for an existing refcat. The rest of this example refers to lsst-devl (e.g. in the GaiaSource path, and the number of cores to use in the convert), so I don't think it hurts to do the same here.

parejkoj · 2021-10-20T20:39:57Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+This is an example script that constructs a conversion lookup table for all converted FITS files in the `refcat_dir` directory, outputting to the `out_dir` directory:
+
+.. code-block:: python
+


Lets put a docstring at the top:

"""Generate an astropy-readable .ecsv files for `butler ingest-files`, to ingest an existing gen2 refcat. """

parejkoj · 2021-10-21T19:16:17Z

doc/lsst.meas.algorithms/creating-a-reference-catalog.rst

+    out_dir = "/path/to/my/output/directory"
+
+    out_file = f"{out_dir}/{os.path.basename(refcat_dir)}.ecsv"
+    print(f"Saving to: {out_file}")


Oh, and I'd suggest putting this after the table.write as a "Wrote output to:" message, to signify that it's done.

leeskelvin · 2021-10-21T20:44:12Z

Thanks John. I think I've addressed everything above. I hope that this now reads well; let me know.

parejkoj

Thanks, this looks good. The links work in the built version.

parejkoj · 2021-10-22T22:01:44Z

Additional comment: we should probably make it clear in section 5 exactly what parts of the commands need to change. Maybe add something like this after the current final sentence:

In particular, you need to change the name of the registered dataset type to "gaia_dr2_20200414" (the refcat used in the python code block above), and the filename to the generated .ecsv file:

... prompt: bash
    butler register-dataset-type REPO gaia_dr2 SimpleCatalog htm7
    butler ingest-files -t direct REPO gaia_dr2 refcats gaia/filename_to_htm.ecsv

leeskelvin · 2021-10-22T22:37:15Z

Final section now reads:

Once this script is complete, finish reference catalog ingestion by following the :ref:`file ingestion instructions above <lsst.meas.algorithms-refcat-ingest>`.
In particular, you need to change the name of the registered dataset type to "gaia_dr2_20200414" (the reference catalog used in the Python code block above), and the filename to the generated .ecsv file ("gaia_dr2_20200414.ecsv" in the Python code block above):

... prompt: bash
    butler register-dataset-type REPO gaia_dr2_20200414 SimpleCatalog htm7
    butler ingest-files -t direct REPO gaia_dr2_20200414 refcats gaia_dr2_20200414.ecsv

parejkoj reviewed Oct 18, 2021

View reviewed changes

parejkoj requested changes Oct 18, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-30624 branch from 76fd317 to ffe1cc0 Compare October 20, 2021 00:28

parejkoj requested changes Oct 20, 2021

View reviewed changes

parejkoj reviewed Oct 21, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-30624 branch 2 times, most recently from b158aa6 to ae0c667 Compare October 21, 2021 20:42

parejkoj approved these changes Oct 21, 2021

View reviewed changes

Document gen2 to gen3 refcat ingestion

74027d4

leeskelvin force-pushed the tickets/DM-30624 branch from ae0c667 to 74027d4 Compare October 23, 2021 00:32

leeskelvin merged commit c98d3c8 into master Oct 23, 2021

leeskelvin deleted the tickets/DM-30624 branch October 23, 2021 00:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DM-30624: Document gen2 to gen3 refcat ingestion #260

DM-30624: Document gen2 to gen3 refcat ingestion #260

leeskelvin commented Oct 14, 2021

parejkoj Oct 18, 2021 •

edited

leeskelvin Oct 19, 2021

parejkoj left a comment

parejkoj left a comment

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

leeskelvin Oct 21, 2021 •

edited

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

parejkoj Oct 20, 2021

parejkoj Oct 21, 2021

leeskelvin commented Oct 21, 2021

parejkoj left a comment

parejkoj commented Oct 22, 2021 •

edited

leeskelvin commented Oct 22, 2021 •

edited


		.. _lsst.meas.algorithms-refcat-existing:

		5. What to do with existing reference catalogs?


		table.write(out_file)

		Once this script is complete, finalize the reference catalog ingestion by following the `file ingestion instructions above <lsst.meas.algorithms-refcat-ingest>`.

		This is an example script that constructs a conversion lookup table for all converted FITS files in the `refcat_dir` directory, outputting to the `out_dir` directory:

		.. code-block:: python

DM-30624: Document gen2 to gen3 refcat ingestion #260

DM-30624: Document gen2 to gen3 refcat ingestion #260

Conversation

leeskelvin commented Oct 14, 2021

parejkoj Oct 18, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parejkoj left a comment

Choose a reason for hiding this comment

parejkoj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leeskelvin Oct 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leeskelvin commented Oct 21, 2021

parejkoj left a comment

Choose a reason for hiding this comment

parejkoj commented Oct 22, 2021 • edited

leeskelvin commented Oct 22, 2021 • edited

parejkoj Oct 18, 2021 •

edited

leeskelvin Oct 21, 2021 •

edited

parejkoj commented Oct 22, 2021 •

edited

leeskelvin commented Oct 22, 2021 •

edited