Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tickets/dm 6352 #36

Merged
merged 6 commits into from
Jul 27, 2016
Merged

Tickets/dm 6352 #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7422cbc
Add an HTM ref cat and use it
r-owen Jul 21, 2016
9f0963b
Update testAstrometryTask.py to use HTM ref cat
r-owen Jul 21, 2016
0c85308
Update testJoinMatchListWithCatalog
r-owen Jul 21, 2016
a07b79c
Update testMatchOptimisticB to not use a.net
r-owen Jul 21, 2016
5f26a28
Remove unused astrometry.net data
r-owen Jul 21, 2016
3831f94
Remove unwanted imports in an sdssrefcat config file
r-owen Jul 22, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ python/lsst/meas/astrom/sip/sipLib_wrap.*
python/lsst/meas/astrom/*Lib.py
python/lsst/meas/astrom/*Lib_wrap.*
tests/.tests
tests/data
tests/testConvertCheby
tests/testlsf1d
tests/testlsf2d
Expand Down
5 changes: 0 additions & 5 deletions tests/astrometry_net_data/no-tag/andConfig.py
View file
Open in desktop

This file was deleted.

584 changes: 0 additions & 584 deletions tests/astrometry_net_data/no-tag/index-photocal-notag.fits
View file
Open in desktop

This file was deleted.

16 changes: 16 additions & 0 deletions tests/data/sdssrefcat/README.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Go to SDSS SkyServer using a web browser
Go to the SQL query page
Select "FITS" as the output format
Run this query:
select objid as id, ra, dec, type, u, g, r, i, z, err_u as u_err, err_g as g_err, err_r as r_err, err_i as i_err, err_z as z_err from PhotoPrimary where ra between 215.1 and 216.1 and dec between 52.6 and 53.6 order by id
Your browser will download a file; rename it to photocal.fits
Make a copy replacing "type" with a "resolved" flag, which is set for objects that are not stars (SDSS type != 6);
"photometric" is set for stars (SDSS type == 6) that have high S/N (r_err < 0.1). This uses "fitscopy",
which is a cfitsio example program that is easy to download and install:
fitscopy photocal.fits"[col id;ra;dec;u;g;r;i;z;u_err;g_err;r_err;i_err;z_err;resolved=(type!=6);photometric=(type==6 && r_err<0.1)]" photocal2.fits
Set up pipe_tasks
Ingest the catalog:
ingestReferenceCatalog.py . photocal2.fits --configfile ingestReferenceCatalogOverride.py
Check the resulting config file
data/sdssrefcat/config/IngestIndexedReferenceTask.py
and remove unwanted imports, if present (see ticket DM-7002)
1 change: 1 addition & 0 deletions tests/data/sdssrefcat/_mapper
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
lsst.obs.test.testMapper.TestMapper
44 changes: 44 additions & 0 deletions tests/data/sdssrefcat/config/IngestIndexedReferenceTask.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import lsst.meas.algorithms.ingestIndexReferenceTask
assert type(config)==lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceConfig, 'config is of type %s.%s instead of lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceConfig' % (type(config).__module__, type(config).__name__)

# Name of column stating if satisfactory for photometric calibration (optional).
config.is_photometric_name='photometric'

# Default HTM level. Level 8 gives ~0.08 sq deg per trixel.
config.level=8

# Name of RA column
config.ra_name='ra'

# Name of Dec column
config.dec_name='dec'

import lsst.meas.algorithms.readFitsCatalogTask
config.file_reader.retarget(target=lsst.meas.algorithms.readFitsCatalogTask.ReadFitsCatalogTask, ConfigClass=lsst.meas.algorithms.readFitsCatalogTask.ReadFitsCatalogConfig)
# HDU containing the desired binary table, 0-based but a binary table never occurs in HDU 0
config.file_reader.hdu=1

# Mapping of input column name: output column name; each specified column must exist, but additional columns in the input data are written using their original name.
config.file_reader.column_map={}

# Name of column stating if the object is resolved (optional).
config.is_resolved_name='resolved'

# Name of column to use as an identifier (optional).
config.id_name='id'

# The values in the reference catalog are assumed to be in AB magnitudes. List of column names to use for photometric information. At least one entry is required.
config.mag_column_list=['u', 'g', 'r', 'i', 'z']

# A map of magnitude column name (key) to magnitude error column (value).
config.mag_err_column_map={'i': 'i_err', 'r': 'r_err', 'u': 'u_err', 'z': 'z_err', 'g': 'g_err'}

# String to pass to the butler to retrieve persisted files.
config.ref_dataset_name='cal_ref_cat'

# Extra columns to add to the reference catalog.
config.extra_col_names=[]

# Name of column stating if the object is measured to be variable (optional).
config.is_variable_name=None

9 changes: 9 additions & 0 deletions tests/data/sdssrefcat/ingestReferenceCatalogOverride.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from lsst.meas.algorithms.readFitsCatalogTask import ReadFitsCatalogTask
config.file_reader.retarget(ReadFitsCatalogTask)
config.ra_name = "ra"
config.dec_name = "dec"
config.mag_column_list = ["u", "g", "r", "i", "z"]
config.mag_err_column_map = {"u":"u_err", "g":"g_err", "r":"r_err", "i":"i_err", "z":"z_err"}
config.is_photometric_name = "photometric"
config.is_resolved_name = "resolved"
config.id_name = "id"
1 change: 1 addition & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903253.fits
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903269.fits
View file
Open in desktop

Large diffs are not rendered by default.

256 changes: 256 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903281.fits
View file
Open in desktop

Large diffs are not rendered by default.

706 changes: 706 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903282.fits
View file
Open in desktop

Large diffs are not rendered by default.

171 changes: 171 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903283.fits
View file
Open in desktop

Large diffs are not rendered by default.

198 changes: 198 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903285.fits
View file
Open in desktop

Large diffs are not rendered by default.

326 changes: 326 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903286.fits
View file
Open in desktop

Large diffs are not rendered by default.

65 changes: 65 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903287.fits
View file
Open in desktop

Large diffs are not rendered by default.

327 changes: 327 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903289.fits
View file
Open in desktop

Large diffs are not rendered by default.

291 changes: 291 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903292.fits
View file
Open in desktop

Large diffs are not rendered by default.

816 changes: 816 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903293.fits
View file
Open in desktop

Large diffs are not rendered by default.

788 changes: 788 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903294.fits
View file
Open in desktop

Large diffs are not rendered by default.

817 changes: 817 additions & 0 deletions tests/data/sdssrefcat/photo_astro_ref/903295.fits
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions tests/data/sdssrefcat/photo_astro_ref/master_schema.fits
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SIMPLE = T / file does conform to FITS standard BITPIX = 16 / number of bits per data pixel NAXIS = 0 / number of data axes EXTEND = T / FITS dataset may contain extensions COMMENT FITS (Flexible Image Transport System) format is defined in 'AstronomyCOMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H END XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional binary table NAXIS1 = 117 / width of table in bytes NAXIS2 = 0 / number of rows in table PCOUNT = 0 / size of special data area GCOUNT = 1 / one data group (required keyword) TFIELDS = 16 / number of fields in each row TTYPE1 = 'flags ' / bits for all Flag fields; see also TFLAGn TFORM1 = '2X ' / format of field FLAGCOL = 1 / Column number for the bitflags. TTYPE2 = 'id ' / unique ID TFORM2 = '1K ' / format of field TDOC2 = 'unique ID' TCCLS2 = 'Scalar ' / Field template used by lsst.afw.table TTYPE3 = 'coord_ra' / position in ra/dec TFORM3 = '1D ' / format of field TDOC3 = 'position in ra/dec' TCCLS3 = 'Angle ' / Field template used by lsst.afw.table TTYPE4 = 'coord_dec' / position in ra/dec TFORM4 = '1D ' / format of field TDOC4 = 'position in ra/dec' TCCLS4 = 'Angle ' / Field template used by lsst.afw.table TTYPE5 = 'parent ' / unique ID of parent source TFORM5 = '1K ' / format of field TDOC5 = 'unique ID of parent source' TCCLS5 = 'Scalar ' / Field template used by lsst.afw.table TTYPE6 = 'u_flux ' TFORM6 = '1D ' / format of field TCCLS6 = 'Scalar ' / Field template used by lsst.afw.table TTYPE7 = 'g_flux ' TFORM7 = '1D ' / format of field TCCLS7 = 'Scalar ' / Field template used by lsst.afw.table TTYPE8 = 'r_flux ' TFORM8 = '1D ' / format of field TCCLS8 = 'Scalar ' / Field template used by lsst.afw.table TTYPE9 = 'i_flux ' TFORM9 = '1D ' / format of field TCCLS9 = 'Scalar ' / Field template used by lsst.afw.table TTYPE10 = 'z_flux ' TFORM10 = '1D ' / format of field TCCLS10 = 'Scalar ' / Field template used by lsst.afw.table TTYPE11 = 'i_fluxSigma' TFORM11 = '1D ' / format of field TCCLS11 = 'Scalar ' / Field template used by lsst.afw.table TTYPE12 = 'r_fluxSigma' TFORM12 = '1D ' / format of field TCCLS12 = 'Scalar ' / Field template used by lsst.afw.table TTYPE13 = 'u_fluxSigma' TFORM13 = '1D ' / format of field TCCLS13 = 'Scalar ' / Field template used by lsst.afw.table TTYPE14 = 'z_fluxSigma' TFORM14 = '1D ' / format of field TCCLS14 = 'Scalar ' / Field template used by lsst.afw.table TTYPE15 = 'g_fluxSigma' TFORM15 = '1D ' / format of field TCCLS15 = 'Scalar ' / Field template used by lsst.afw.table TFLAG1 = 'photometric' TFLAG2 = 'resolved' TTYPE16 = 'footprint' / archive ID for Footprint TFORM16 = '1J ' / format of field TDOC16 = 'archive ID for Footprint' TCCLS16 = 'Scalar ' / Field template used by lsst.afw.table AR_HDU = 3 / HDU containing the archive index for non-recordHIERARCH AFW_TABLE_VERSION = 1 AFW_TYPE= 'SOURCE ' / Tells lsst::afw to load this as a Source table.END XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional binary table NAXIS1 = 148 / width of table in bytes NAXIS2 = 0 / number of rows in table PCOUNT = 0 / size of special data area GCOUNT = 1 / one data group (required keyword) TFIELDS = 7 / number of fields in each row TTYPE1 = 'id ' / Archive ID of the persistable object that owns TFORM1 = '1J ' / format of field TDOC1 = 'Archive ID of the persistable object that owns the records pointed &'CONTINUE 'at by this entry' TCCLS1 = 'Scalar ' / Field template used by lsst.afw.table TTYPE2 = 'cat.archive' / index of the catalog this entry points to, fromTFORM2 = '1J ' / format of field TDOC2 = 'index of the catalog this entry points to, from the perspective of &'CONTINUE 'the archive' TCCLS2 = 'Scalar ' / Field template used by lsst.afw.table TTYPE3 = 'cat.persistable' / index of the catalog this entry points to, fromTFORM3 = '1J ' / format of field TDOC3 = 'index of the catalog this entry points to, from the perspective of &'CONTINUE 'the Persistable' TCCLS3 = 'Scalar ' / Field template used by lsst.afw.table TTYPE4 = 'row0 ' / first row used by the persistable object in thiTFORM4 = '1J ' / format of field TDOC4 = 'first row used by the persistable object in this catalog' TCCLS4 = 'Scalar ' / Field template used by lsst.afw.table TTYPE5 = 'nrows ' / number of rows used by the persistable object iTFORM5 = '1J ' / format of field TDOC5 = 'number of rows used by the persistable object in this catalog' TCCLS5 = 'Scalar ' / Field template used by lsst.afw.table TTYPE6 = 'name ' / unique name for the persistable object's class TFORM6 = '64A ' / format of field TDOC6 = 'unique name for the persistable object''s class' TCCLS6 = 'String ' / Field template used by lsst.afw.table TTYPE7 = 'module ' / Python module that should be imported to registTFORM7 = '64A ' / format of field TDOC7 = 'Python module that should be imported to register the object''s fac&'CONTINUE 'tory ' TCCLS7 = 'String ' / Field template used by lsst.afw.table EXTTYPE = 'ARCHIVE_INDEX' AR_CATN = 0 / # of this catalog relative to the start of thisAR_NCAT = 1 / # of catalogs in this archive, including the inHIERARCH AFW_TABLE_VERSION = 1 END