New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Develop Database Schema #6
Comments
how flexible is this? the JWST keywords, header info, filetypes, etc. are still in flux (nowhere near as stable as WFC3), so we need to be able to evolve the schema in response to these changes. |
If we build this right, changes to the schema for the header tables should be as simple as updating a text file and adding/removing columns in the database. Changes to the data structure itself (i.e. new filetypes, new/different FITS extensions) would be a bit trickier because that would mean adding new tables and not just new columns. This brings up another question: How often should we anticipate changes to the header keywords/filetypes/FITS extensions after launch? |
My guess is that header keyword changes after launch won't be too common, but I'm sure it will happen from time to time. For what it's worth, I have a function that returns all of the header keywords for a requested reference file type. It does this by reading in the appropriate schema definition files that SSB has in the JWST Calibration Pipeline repo. I doubt it would be hard to update it to work on the data filetypes. |
Filetypes that will be ingested into MAST:
I'll put together more details on each soon. |
Data structures:JWST jargon
_uncal.fits - raw, uncalibrated file
_rate.fits - countrate images (equivalent to HST flt)This is the output of the Level 2A pipeline, which includes basic calibrations (superbias subtraction, linearity correction, slope fitting). For an exposure that contains a single integration the
*_cal.fits - Calibrated fileOutput from level 2b pipeline. Flux calibration applied, flat field applied, distortion solution added. Similar to the
Extensions are the same as in the case of the |
Thanks @bhilbert4 this is very helpful! |
JDox page on filetypes and formats: |
i2d.fits file format - Identical to _cal.fits format above.
|
just to confirm, based on Tom Donaldson's confluence page and what @bhilbert4 has said here, if you have (for ex) a *_rate.fits and a *_uncal.fits that both correspond to the same original image, everything in the |
My understanding is that the rest of the filename will be consistent between the two. |
Yes, that's correct |
Per @SaraOgaz Jonathon pointed me to this doc page for the pipeline where there’s a whole section about the associations: https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/index.html |
Now that we have decided to the use MAST api, this is no longer needed. |
We need to develop a schema for the
jwql
database. I think a decent starting point is something like the schema I used for ACS Quicklook:In this schema, we have a
master
table that keeps track of each rootname that is in the database and when it was ingested. Thedatasets
table keeps track of which filetypes exist for a given rootname. Then there is a table for eachdetector
/extension
/filetype
combination which is basically a dump of the headers (columns are header keys and values are header values).To construct this for
jwql
, we will need to know the following for each instrument:The text was updated successfully, but these errors were encountered: