processed HSC data for jointcal tickets/DM-8555

README.md

@@ -1,7 +1,7 @@
 jointcal test data
 ==================
 
-This repository contains data to test the performance of the [jointcal](http://github.com/lsst/jointcal) product.
+This repository contains data to test the performance of the [jointcal](http://github.com/lsst/jointcal) product. Test data for jointcal needs to have multiple visits of the same field, processed with the LSST DM stack to get VisitInfo metadata and source catalogs.
 
 Individual sets of testing data should be placed in their own directories, as butler-accessible repositories. They can then be loaded in testing code via the butler and the catalogs fed to jointcal. The image HDUs in the calexps in each repo have been set to identically zero and then compressed to save space, since the calexps are only needed for the calexp_md metadata.
 
@@ -12,7 +12,7 @@ twinkles1
 
 Twinkles is a synthetic survey produced for the Dark Energy Science Collaboration's first data challenge. The intent with Twinkles is to test detection and analysis of type Ia SNe and strongly lensed quasars. Stars, galaxies and solar system objects were drawn from the LSST universe model. In addition to the standard universe, higher than natural rates of both type Ia SNe and lensed quasar systems were planted in the images. This repository includes one chip for the first 10 r-band visits in a deep drilling field (all observations are on the same night). See [here](https://github.com/DarkEnergyScienceCollaboration/Twinkles/blob/master/doc/Design.md) for a more complete description of the full survey.
 
-**Note:** the calexps contained in this repository are identically zero and compressed and are only included for their metadata to build the `calexp_md`. If you want to use the twinkles images, please use the link above for details.
+The calexps contained in this repository are identically zero and compressed and are only included for their metadata to build the `calexp_md` and VisitInfo. If you want to use the twinkles images, please see the link above for details.
 
 twinkles1_and_index
 -------------------
@@ -22,7 +22,7 @@ Astrometry index catalogs corresponding to the twinkles1 data.
 cfht
 ----
 
-Source catalogs, metadata, and zeroed+compressed images derived from `validation_data_cfht`, processed with a version of the stack that provides VisitInfo (afw post-`tickets/DM-8052` merge: git hash 22f008f).
+Source catalogs, metadata, and zeroed+compressed images derived from [validation_data_cfht](https://github.com/lsst/validation_data_cfht), processed with a version of the stack that provides VisitInfo (afw post-`tickets/DM-8052` merge: git hash 22f008f).
 
 Then the `icExp/`, `icSrc/`, and `srcMatch` directories were deleted from `cfht/` and the calexps were processed via the included `compress_jointcal_cfht_test_data.py` to remove the pixel-level data and gzip compress them to make their size reasonable. I saved a skyMap to deepCoadds, generated by running:
 
@@ -50,9 +50,9 @@ Astrometry index catalogs corresponding to the cfht data.
 decam
 -----
 
-Source catalogs, metadata, and zeroed+compressed images derived from `validation_data_decam`, processed with a version of the stack that provides VisitInfo (afw post-`tickets/DM-8052` merge: git hash 22f008f).
+Source catalogs, metadata, and zeroed+compressed images derived from [validation_data_decam](https://github.com/lsst/validation_data_decam), processed with a version of the stack that provides VisitInfo (afw post-`tickets/DM-8052` merge: git hash 22f008f).
 
-The `bkgd/`, `icMatch/`, and `icSrc/` directories were deleted from the visit sub-directories and the calexps were processed via the included `compress_jointcal_decam_test_data.py` to remove the pixel-level data and gzip compress them to make their size reasonable. I saved a skyMap to `deepCoadds/`, generated by running:
+The `bkgd/`, `icMatch/`, and `icSrc/` directories were deleted from the visit sub-directories and the calexps were processed via the included `compress_jointcal_decam_test_data.py` to remove the pixel-level data and gzip compress them to make their size reasonable. The source catalogs were gzipped to reduce their size. I saved a skyMap to `deepCoadds/`, generated by running:
 
 ```
 makeDiscreteSkyMap.py cfht --output data/deepCoadd --id visit=0176837^0176846 --configfile makeSkyMapConfig.py
@@ -76,6 +76,48 @@ decam_and_index
 
 Astrometry index catalogs corresponding to the decam data, copied from `validation_data_decam/astrometry_net_data/`.
 
+hsc
+---
+
+Source catalogs, metadata, and zeroed+compressed images derived from an earlier non-full-focalplane version of [validation_data_hsc](https://github.com/lsst/validation_data_hsc) ([git hash: f20a3ec](https://github.com/lsst/validation_data_hsc/commit/f20a3ec9ab1e17b40f46429711ef2d185a4d6596), processed with a version of the stack that provides VisitInfo (afw post-`tickets/DM-8052` merge: git hash 22f008f). This data includes two filters (R and I) with 11 total visits, 2-4 CCDs per visit from different parts of the focal plane.
+
+The commands that were run to produce this repo, from within the `DATA` directory of a clone of the validation_data_hsc repo at the commit linked above:
+
+```bash
+setup testdata_jointcal
+setup obs_subaru
+setup pipe_drivers
+
+export OMP_NUM_THREADS=1
+
+export SETUP_ASTROMETRY_NET_DATA="astrometry_net_data sdss-dr9-fink-v5b"
+export ASTROMETRY_NET_DATA_DIR=`pwd`/sdss-dr9-fink-v5b
+
+# change this directory to where you want the processed data to live
+export OUTPUT=$TESTDATA_JOINTCAL_DIR/hsc
+echo lsst.obs.hsc.HscMapper > $OUTPUT/_mapper
+ingestImages.py $OUTPUT --mode=link 'STRIPE82L/*/*/HSC-?/*.fits'
+
+# older few-chip-per-visit data
+VISIT=903334^903336^903338^903342^903344^903346^903986^903988^903990^904010^904014
+
+singleFrameDriver.py $OUTPUT --calib CALIB --output $OUTPUT --job singleFrame --cores 4 --id visit=$VISIT --clobber-config --clobber-versions
+makeDiscreteSkyMap.py $OUTPUT --output $OUTPUT --id visit=$VISIT --clobber-versions
+```
+
+The following directories and files were deleted, and the calexps were processed via the included `compress_jointcal_hsc_test_data.py` to remove the pixel-level data and gzip compress them to make their size reasonable. The source catalogs were gzipped to reduce their size.
+
+```
+hsc/00*/HSC-?/output/ICSRC-*.fits
+hsc/00*/HSC-?/thumbs/
+hsc/00*/HSC-?/corr/BKGD-*
+hsc/00*/HSC-?/corr/ICEXP*
+```
+
+hsc_and_index
+-------------
+
+Astrometry index catalogs corresponding to the decam data, copied from `validation_data_hsc/astrometry_net_data/`.
 
 Git LFS
 -------

compress_jointcal_hsc_test_data.py

@@ -0,0 +1,20 @@
+#!/usr/bin/env python
+"""Zero out the imageHDUs in the calexps, and then gzip the files."""
+from __future__ import absolute_import, division, print_function
+
+from astropy.io import fits
+import glob
+import subprocess
+
+files = glob.glob('hsc/00*/HSC-?/corr/CORR-*.fits')
+for f in files:
+    data = fits.open(f)
+    print('processing:', f)
+    for hdu in data:
+        if isinstance(hdu, fits.ImageHDU):
+            hdu.data[:] = 0
+    data.writeto(f, clobber=True)
+    # gzip the file
+    subprocess.call(['gzip', f])
+    # move it back to the original .fits filename to keep the butler happy.
+    subprocess.call(['mv', f + '.gz', f])