Remove dependencies on the Legacy Surveys Gaia astrometry files

[geordie666]

This PR removes our reliance on Legacy Surveys chunked "astrom" files when handling Gaia data. Changes include:

• Using a $GAIA_DIR environment variable instead of passing a Gaia directory around.
• New functions to retrieve and organize Gaia files in the $GAIA_DIR. Including:
  • A function to wget Gaia DR2 CSV files from the ESA archive.
  • A function to convert the Gaia DR2 CSV files to FITS.
  • A function to reorganize the Gaia FITS files into (NESTED) HEALPixels.
• Using the NESTED HEALPix scheme for Gaia files throughout desitarget (the Legacy Surveys "astrom" files used the RING scheme).
• Renaming the output column TYPE to MORPHTYPE for GFA targets.

[geordie666]

For the record, example output files are at NERSC in:

/project/projectdirs/desi/target/gaia_dr2/csv
/project/projectdirs/desi/target/gaia_dr2/fits
/project/projectdirs/desi/target/gaia_dr2/healpix

Which correspond to setting the $GAIA_DIR environment variable to /project/projectdirs/desi/target/gaia_dr2.

[geordie666]

Also, this should fix #437.

[moustakas]

This is fabulous, @geordie666. May I make a request for a data file that directly pertains to this PR and which would be very helpful for a higher level of realism in select_mock_targets? The (FITS) file I would like is a healpix map of the (Gaia-derived) stellar density over the whole sky, which I could then use to modulate the stellar contamination rate with Galactic coordinates. Having a simple function (like what you wrote for the pixel weights) to downsample / upsample from the nominal nside (e.g., 128 or 256) to another nside would also be very useful.

If this makes sense and is something you’d be willing to write and include here that’d be awesome. If not, no worries—it’s definitely something I can tackle separately.

[sbailey]

@moustakas please put your request into a new issue. Since we're about to lose @geordie666 for the holidays, I'm eager to get this merged and tagged so that we can get a new TS run to go along with 18.12.

[geordie666]

@moustakas: I think what you requested already exists in the pixweight- files. The STARDENS columns in those files is agnostic to the Data Release (it's determined for the whole sky). For instance:

a = fitsio.read("/project/projectdirs/desi/target/catalogs/dr7.1/0.22.0/pixweight-dr7.1-0.22.0.fits")
b = fitsio.read("/project/projectdirs/desi/target/catalogs/dr6/0.22.0/pixweight-dr6.0-0.22.0.fits")
set(a["STARDENS"]-b["STARDENS"])
{0.0}

That's derived from the Gaia density in the randoms.py code with some restrictions to derive only "stellar" sources:

https://github.com/desihub/desitarget/blob/master/py/desitarget/randoms.py#L531-L577

As a bonus, I think the code to upsample/downsample those pixweight files is already in desitarget.io.load_pixweight_recarray:

https://github.com/desihub/desitarget/blob/master/py/desitarget/io.py#L907

Let me know if you need something else, though, and open a post-holidays issue for me.