Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Updates for Release 19.9 #78

Merged
merged 2 commits into from Dec 4, 2019
Merged

[WIP] Updates for Release 19.9 #78

merged 2 commits into from Dec 4, 2019

Conversation

weaverba137
Copy link
Member

This PR updates the data model for release 19.9. This is not ready to merge yet, but please use this PR for comments about the data model for release 19.9.

@weaverba137 weaverba137 added the WIP Work in Progress label Oct 2, 2019
@weaverba137 weaverba137 self-assigned this Oct 2, 2019
@weaverba137
Copy link
Member Author

Questions and Problems from 19.9

@sbailey please review!

DESISURVEY_OUTPUT

  • No model file for exposures.fits, test-tiles.fits.

DESI_SPECTRO_DATA

  • The datachallenge files are not a useful comparison to the model.
  • See also comments on fibermap files below.

DESI_SPECTRO_REDUX

  • Many files types are not generated by the datachallenge.
  • I'm very worried that passing around huge fibermap/target tables is going to
    result in lots of inconsistencies between files as columns are added or
    removed.

DESI_SPECTRO_SIM

  • No comparison possible for pix and simpix files.
  • Only routine header keyword and column name cleanup are needed.
  • Fibermap files no longer have HDU2, TARGETS.
  • Simspec TRUTH table no longer has CONTAM_TARGET.
  • Why are NUMOBS_INIT and PRIORITY_INIT int64, when NUMOBS_MORE is int32.
    And in any case, why are we planning for 2**32 or 2**64 exposures?
  • There are a number of type changes in simspec, and some columns
    that used to have units no longer do. See the OBSCONDITIONS table especially.

DESI_TARGET

  • Is there any way to more accurately represent the "official" pipeline layout?
    For example, is the position of fiberassign under DESI_TARGET reflective of
    actual operations?
  • The fiberassign data model file is named tile-TILEID-FIELDNUM.rst but the files
    are named tile-TILEID.fits.
  • What happened to the standards files?
  • Why did skies.rst get named that? It's called sky.fits.
    Are these really meant to be the same? There are a lot of changes.
  • mtl files have no units on any column. Some of those columns have to have units.
    In fact, units are missing from basically every DESI_TARGET file.
  • mtl files have NUMOBS_INIT as int64, but it is int32 in other files.
  • targets file do not contain PRIORITY?
  • There are several minor differences among truth, sky, mtl that should be
    checked in detail.
  • Why are ALLMASK_[GRZ] columns float32? The description is pretty clear
    that it should be an integer.
  • SEED keyword in truth table looks dubious. Why not an integer with a comment?
  • Layout of tile file FIBERASSIGN table is radically different from other
    files.
  • Does every HDU in tile have to have the same keywords?

update .travis.yml

update setup.py and tests

tweak copyright dates

fix name error

fix name of stats file

fix name of exposures file

working on exposures_surveysim

remove extname keywords

remove extname keywords

remove comment

update minitest script

fix filenames in DESI_TARGET

fix filenames in DESI_TARGET

remove QUICKLOOK

remove link to QUICKLOOK

working on simspec

working on simspec

skip TNULL keywords

extra column

working on mtl

working on mtl

working on sky

working on sky

working on sky

working on targets

working on targets

working on targets

working on targets

working on truth [skip ci]

working on tile [skip ci]

working on tile [skip ci]

working on tile [skip ci]

working on tile [skip ci]

update logic for extname in hdu 0

fix table border

working on zcatalog [skip ci]

working on sky [skip ci]

working on sky [skip ci]

working on calib [skip ci]

working on cframe [skip ci]

working on cframe [skip ci]

working on frame [skip ci]

working on spectra [skip ci]

working on spectra [skip ci]

working on spectra [skip ci]

working on zbest [skip ci]

working on zbest [skip ci]
@sbailey
Copy link
Contributor

sbailey commented Dec 4, 2019

Thanks for this review and update. There are a large number of open questions here, but I am going to merge this PR as progress and move the questions into a ticket for future work.

@sbailey sbailey merged commit 007b017 into master Dec 4, 2019
@sbailey sbailey deleted the 19.9 branch December 4, 2019 22:12
@sbailey sbailey mentioned this pull request Dec 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WIP Work in Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants