New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workaround for bad flux values #722
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine (though I haven't tested it myself).
I think this is ready to merge. I tested this with the redshift-database tutorial, which itself needed updates for 18.11++. |
This fixes the bad flux values, but I'm still getting a crash from loading a full minitest production: basedir = '/global/cscratch1/sd/sjbailey/desi/dev/end2end'
targetdir = '/global/cscratch1/sd/sjbailey/desi/dev/end2end/targets'
reduxdir = '/global/cscratch1/sd/sjbailey/desi/dev/end2end/spectro/redux/mini'
fibassigndir = '/global/cscratch1/sd/sjbailey/desi/dev/end2end/fiberassign'
expfile = '/global/cscratch1/sd/sjbailey/desi/dev/end2end/survey/exposures.fits'
from argparse import Namespace
from desispec.database.redshift import (setup_db, dbSession, load_file, ObsList, Target, Truth, ZCat, FiberAssign,
load_fiberassign)
options = Namespace(overwrite=True, dbfile=os.path.join(os.getenv('SCRATCH'), 'minitest.db'), hostname=None, maxrows=0,
chunksize=50000, schema=None, username=None, verbose=False, datapath=basedir)
# We'll be using a SQLite database, ignore the return value of setup_db
postgresql = setup_db(options)
load_file(expfile, ObsList, hdu='EXPOSURES', expand={'PASS': 'passnum'})
load_file(os.path.join(targetdir, 'truth.fits'), Truth, hdu='TRUTH')
load_file(os.path.join(targetdir, 'targets.fits'), Target, hdu="TARGETS")
load_file(os.path.join(reduxdir, 'zcatalog-mini.fits'), ZCat, hdu='ZCATALOG',
expand={'COEFF': ('coeff_0', 'coeff_1', 'coeff_2', 'coeff_3', 'coeff_4',
'coeff_5', 'coeff_6', 'coeff_7', 'coeff_8', 'coeff_9',)},
rowfilter=lambda x: ((x['TARGETID'] != 0) & (x['TARGETID'] != -1)))
load_fiberassign(fibassigndir) Results in
|
@sbailey, you need to look at the redshift-database tutorial to see how to change the loading sequence. |
@weaverba137 got it. For the record, this worked. I'll merge now and update the minitest notebook. from argparse import Namespace
from desispec.database.redshift import (setup_db, dbSession, load_file, ObsList, Target, Truth, ZCat, FiberAssign,
load_fiberassign, update_truth)
options = Namespace(overwrite=True, dbfile=os.path.join(basedir, 'minitest.db'), hostname=None, maxrows=0,
chunksize=50000, schema=None, username=None, verbose=False, datapath=basedir)
# We'll be using a SQLite database, ignore the return value of setup_db
postgresql = setup_db(options)
print('--- Loading exposure list')
load_file(expfile, ObsList, hdu='EXPOSURES', expand={'PASS': 'passnum'})
print('--- Loading truth tables')
load_file(os.path.join(targetdir, 'truth.fits'), Truth, hdu='TRUTH')
for h in ('BGS', 'ELG', 'LRG', 'QSO', 'STAR', 'WD'):
update_truth(os.path.join(targetdir, 'truth.fits'), 'TRUTH_' + h)
print('--- Loading targets')
load_file(os.path.join(targetdir, 'targets.fits'), Target, hdu='TARGETS',
expand={'DCHISQ': ('dchisq_psf', 'dchisq_rex', 'dchisq_dev', 'dchisq_exp', 'dchisq_comp',)})
print('--- loading redshift catalog')
load_file(os.path.join(reduxdir, 'zcatalog-mini.fits'), ZCat, hdu='ZCATALOG',
expand={'COEFF': ('coeff_0', 'coeff_1', 'coeff_2', 'coeff_3', 'coeff_4',
'coeff_5', 'coeff_6', 'coeff_7', 'coeff_8', 'coeff_9',)},
rowfilter=lambda x: ((x['TARGETID'] != 0) & (x['TARGETID'] != -1)))
print('--- loading fiber assignments')
load_fiberassign(fibassigndir)
print('--- done') |
This PR is a workaround for desihub/desitarget#397.
Do not merge yet. I also need to review changes to the data model (for example, the target table has added about 60 columns).