NIRSpec MOS level 3 spectral combination issues #3918
Comments
|
Comment by James Davies: Thanks for the report James. If you are manually changing pointing information in keyword headers, that needs to be done on the _rate.fits file, i.e the input to the spec2 pipeline. This is the only time it is used. It is used in the very first Getting the FITS header keywords for pointing populated correctly for simulated data is tricky, as one has to figure out which change in telescope pointing and roll (translated into RA_REF, DEC_REF and ROLL_REF and the aperture reference) will produce the nod you need in your data. I suspect it is especially tricky in NIRSpec given the angle of the detector relative to spacecraft V2 (that is in the V3I_YANG keyword header). If you are editing the pointing info after the data has been extracted into cutouts and written to the _cal.fits file, then you'll have to do it before that on the input to the spec2 pipeline. Of course there could actually be a problem in the resampling code, but it would be good to make sure that the GWCS is being created as intended in |
|
Comment by James Muzerolle: An update on this issue: I reprocessed this same simulated data after having added the correct (I think) spacecraft pointing information to the rate files (using cal version 0.13.7). Both problems reported above are still evident. I have replaced the old files with the new versions in the same directory on central store. |
|
Comment by Jane Morrison: [~muzerol] |
|
Comment by James Muzerolle: Yes, both the level 2 and 3 association files I used are now in the same directory as the data (/grp/jwst/wit4/nirspec/Muzerolle/Files_spec3). Note the background subtraction is done at level 2. I also updated the fits files to the latest versions processed with b7.4, though I don't think there were any substantial changes from the last version. |
|
Comment by Jane Morrison: [~bushouse][~eisenham] [~jdavies] |
|
Comment by Howard Bushouse: Yes, the slit attributes are in the ASDF extension, but it won't work to just do a simple delete of the entire ASDF extension, because you'd also lose the WCS objects for each slit. And you can't simply rerun assign_wcs on the modified cal files (with only 2 slits), because assign_wcs is build to work on the original full 2-D image before the slits have been extracted. So you either need to edit out certain specific parts of the ASDF meta tree or start over completely from the original rate files (if they exist) and redo all of the calwebb_spec2 processing to extract only 2 slits to begin with. [~eisenham] or [~jdavies] might have ideas of how to do specific edits of the ASDF meta tree. |
|
@jmuzerolle |
|
Hi Jane, That's right, I skipped the flat_field step because the simulations don't include the major component of that. Otherwise, I ran calspec2 as normal with the json association file. Let me know if you need help in modifying the MSA metafile, I have code that can re-create it with just a few slits instead of the entire set. |
|
Comment by Jane Morrison: [~muzerol] |
|
Comment by James Muzerolle: [~morrison] I put both the original MSA metafile ("jw95065006001_msa_full.fits") and a new version with only 2 slits/sources ("jw95065006001_msa.fits") in /grp/jwst/wit4/nirspec/Muzerolle/Files_spec3. The MSA metafile contains information on all of the shutters that were commanded open to form the slits corresponding to the set of science sources, which is needed by assign_wcs and extract_2d. The name of the metafile is contained in the primary header keyword "MSAMETFL", and has to be present before calspec2 can be run. You should be able to rerun the F170LP*correctedWCS.fits files through calspec2 without changing anything else. If you want to try different slits, you can edit the script "edit_metafile.py" and change the "saveslits" array to whatever slit IDs you want. Let me know if you have any problems! |
|
Comment by Jane Morrison: [~muzerol] I grabbed the MSA file and tried to reprocess the data. I looked at the MSA file and it is set up |
|
Comment by Jane Morrison: Are these slits only on NRS1 ? Or should they be on both detectors. |
|
Comment by James Muzerolle: Yes, both of those particular slits are only on NRS1. Sorry about that, I thought I had picked one slit that was on both detectors. Try slit 31 for an example that should be roughly evenly on both. I'm surprised the pipeline throws an error in this case, though. Since there may be programs that don't have any spectra on one detector, we should have spec2 exit more gracefully. |
|
Comment by Jane Morrison: Ok I will try slit 31 and I agree with you we should not get a error - but a clean exit. So I will try and fix that too, |
|
Comment by Howard Bushouse: It's supposed to exit more gracefully than that, after issuing the error message about no open slits. Instead of resulting in a traceback it should just say the pipeline is shutting down and then shut down. It can't proceed any further to subsequent steps without any slits to work on. |
|
Comment by Jane Morrison: [~muzerol] I pulled out slit 31 and re-ran spec2 to produce the cal files for NRS1 and NRS2. I put the results of calspec3 for this slit at /grp/jwst/ssb/chartreuse/calspec3/JP-936 |
|
Comment by James Muzerolle: The combined product looks like a resampled version of the 1st nod exposure on NRS1, as if the other 2 nods and the data on NRS2 had not been included. If all nods were being combined, I would expect to see the negative traces on both sides of the combined positive trace. Including both detectors, the wavelength range should be much larger (roughly 1.8 - 3.2 microns), and there should be a large gap roughly in the middle at the wavelengths where the spectrum lay across the detector gap. |
|
Comment by Jane Morrison [X]: [~muzerol] |
|
Comment by Jane Morrison [X]: [~bushouse][~eisenham] [~jdavies] |
|
Comment by Jane Morrison [X]: [~muzerol] |
|
Comment by Jane Morrison [X]: [~muzerol] I grabbed the MSA file and tried to reprocess the data. I looked at the MSA file and it is set up |
|
Comment by Jane Morrison [X]: Are these slits only on NRS1 ? Or should they be on both detectors. |
|
Comment by Jane Morrison [X]: Ok I will try slit 31 and I agree with you we should not get a error - but a clean exit. So I will try and fix that too, |
|
Comment by Jane Morrison [X]: [~muzerol] I pulled out slit 31 and re-ran spec2 to produce the cal files for NRS1 and NRS2. I put the results of calspec3 for this slit at /grp/jwst/ssb/chartreuse/calspec3/JP-936 |
|
Comment by James Muzerolle: The e0 and e1 cutouts on both detectors look about as expected, but e2 looks somewhat blocky with several discontinuities. Are those from the cal files? Here's what I got for that exposure, with no background nod subtraction and using extended slit_y limits of +/-1: !slit31_e2_nrs1_cal.png! !slit31_e2_nrs2_cal.png! Note, the signal at the top is from the extended PSF wing of a source in an adjacent slitlet. The actual source is at the bottom, and is mostly cut off using the default slit_y limits.
|
|
Comment by Jane Morrison: [~muzerol] The two images that show are those from e2 position ? Are they the s2d images or something else ? |
|
Comment by Jane Morrison: Ok here are the cal file images - 3 top ones NRS1 and 3 bottom ones NRS2. Now these do not have |
|
Comment by James Muzerolle: I see, sorry, I thought you were showing unrectified spectra from the cal files. The pictures I attached are the unrectified spectra for the e2 position (I'm using 0.15.1, so it doesn't include the most recent changes you've made to resample_spec). The s2d results you show look reasonable, except e2 is missing most of the (intended) source. Most of what you're seeing in that case is the wings of a source in the adjacent slitlet, resampled into the pixel space of the intended source. If you extend the slit_y limits like I did above you should see the full source spectrum at the bottom. I may need to put in a separate ticket to look more closely at how the slit_y limits are applied in assign_wcs, as it seems many source traces are being cut off if the source position is too close to the top/bottom edge of the slitlet. |
|
Comment by James Muzerolle: Didn't see your last comment until after I posted... real-time conversations are tricky on confluence! :) Your unrectified images look as expected. If you extend the slit_y limits, you should see the source pop out in e2. |
|
Comment by Jane Morrison: Here are the results if I do extend the slit. For e0 it looks like we might be picking up another slit at the bottom when we do this ? These are the cal image (so unrectified) |
|
Comment by Jane Morrison: I was not sure if this image attached to last comment |
|
Comment by James Muzerolle: This looks the same as the previous one? One thing we could check before running calspec3 combination is to calculate RA and Dec per pixel from the WCS object, and see if the source RA & Dec (as given in the CAL SCI header) is located at the position of the source in the individual spectra. |
|
Comment by James Muzerolle: I'll take a look at my cal files to do this... |
|
Comment by Jane Morrison: Here is the correct screen capture (I am having issues attaching images to comments today) |
|
Comment by Jane Morrison: !Screen Shot 2020-04-20 at 12.20.21 PM.png|thumbnail! |
|
Comment by Jane Morrison: [~muzerol] to make more progress on if resample_spec is combining data together correctly is there maybe a different slit we could use. One has the source on the three nods sort of where it is expected. I can then test that the final resample spec image has a positive trace with two negative traces on either side. |
|
Comment by James Muzerolle: [~morrison] The last results you posted for the exposure-level s2d products look reasonably good, at least by-eye. However, I think I may have found a problem with the WCS that would affect the multi-exposure combination - the RA and Dec values at the source trace in the cal files at each nod is offset by roughly 1 arcsec. That means that the source traces won't overlap when resampled onto a common grid. This is only preliminary, though, I need to double-check that I'm calculating the RA and Dec values per pixel correctly. In the meantime, if you want to try a different slit, 34 might be a good choice. You won't need to extend the slit_y limits, and there is no contamination from adjacent sources. Let me know if you need me to send a modified MSA metafile for that.
|
|
Comment by James Muzerolle: [~morrison] There might be a problem with the header keyword values for the spacecraft pointing. I had to calculate them separately and add them in by hand since we have no way of running the simulations through SDP. It's the only thing I can think of that would explain the discrepant RA,Dec values. I'll try to find and correct the offending keywords and send new versions of the fits files asap. |
|
Comment by Jane Morrison: [~muzerol] I did rerun with slit 34 and I think it looks pretty good now. |
|
Comment by Jane Morrison: !Screen Shot 2020-04-22 at 4.07.57 PM.png|thumbnail! |
|
Comment by James Muzerolle: [~morrison] I think I found the problem, "ROLL_REF" was off by 180 degrees. I don't completely understand why, but after making the change the calculated RA, Dec values at the source trace in all nodded exposures are consistent to within a few milliarcsec. I've copied new versions of the count rate and spec2 product files to /grp/jwst/wit4/nirspec/Muzerolle/Files_spec3/. |
|
@jmuzerolle Cool. I will grab the new files and run the new data through calspec3. |
|
Comment by Jane Morrison: [~muzerol] I think this PR is finished. I ran calspec3 using your new data and I put the results of |
|
Comment by James Muzerolle: [~morrison] The directory is empty, except for a cfg and association file. |
|
Comment by Jane Morrison: Sorry about that. I needed to redo it because I used incorrect files. I am re-doing it now. It is taking a while because all the slits are in the cal file. Let you know when it is done. |
|
Comment by Jane Morrison: [~muzerol] I am copying the results from running calspec3 on all the slits. I looked a sampling of s2d images - I am curious if you think they look ok - it sort of looks like there are 3 negative traces in many of the images ? |
|
Comment by James Muzerolle: [~morrison] This is a significant improvement! In an ideal situation where none of the traces are cut off, I would expect to see a single positive trace surrounded by 2 negative traces on either side. In the few s2d files I looked at, I see something close to this, though one or two of the negative traces have been cut off. I think this may be because the output grid spatial range is too small? Interestingly, the best results with the tightest resampled traces seem to be where there is only data on one detector (e.g., slit 417), which may point to some issue with how the output pixel grid is being constructed. For the cases I looked at with data from both detectors, the positive trace seems to be more spread out particularly at wavelengths where the unrectified trace has the most curvature. I also looked at the CON extension for these cases, I don't completely understand the numbers there. The highest values range from as low as ~7 (slit 417) to as high as ~50 (slit 227). I know this is supposed to indicate the overlap among all exposures that went into the result, but it seems odd that the numbers would be so different. I'm just displaying things in fv, perhaps I'm missing some information? I'll try to look more systematically at the results in python later today. |
|
Comment by Jane Morrison: [~muzerol][~jdavies] Before the resample step all the input data that a single slit falls on is grouped together
2 . The spatial sampling is set by the first input in the group. I was assuming that the spatial sampling would
Do you see anything wrong ? I really hope that all made sense. |
|
Comment by James Muzerolle: [~morrison] I don't see anything obviously amiss in your description of how the output grid is determined. Two things we could look at next: verify that the RA,Dec values in the individual cal.fits files are sufficiently in agreement across the entire trace (I had previously only looked at the mean value), and compare the full range of RA,Dec in each with the output grid to see exactly which values are getting cut off. I will take a closer look at this next week. Actually, I could use some help on how to access the RA, Dec, and wavelength values in the level 3 s2d file. I looked at it yesterday but couldn't figure out how to extract the information from the data model. One thing I did notice from the above description - you say that the inputs are grouped together according to slit. This really should be source, because there will be observations that include spectra of the same source taken in different slits; for example, a set of dithered exposures (the simulations we've been using don't have this problem because there is only one set of slits). If the code is currently keying off of SLITID, would it be straightforward to change to SOURCEID instead? |
|
Ok I realize I did use the wrong term to say how the data was grouped together. I am pretty sure it is SOURCE ID - my mistake. |
|
Great, no worries!
…________________________________
From: Jane Morrison <notifications@github.com>
Sent: Thursday, April 30, 2020 6:40 PM
To: spacetelescope/jwst
Cc: James Muzerolle; Mention
Subject: Re: [spacetelescope/jwst] NIRSpec MOS level 3 spectral combination issues (#3918)
Ok I realize I did use the wrong term to say how the data was grouped together. I am pretty sure it is SOURCE ID - my mistake.
I just think of the groupings as a bunch of slits. So I used sloppy terminology in my description.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#3918 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEITZALS65DW36NQCYFHC6TRPH47PANCNFSM4ILYNJZA>.
|
|
Comment by Howard Bushouse: Fixed (at least mostly) in [https://github.com//pull/4766] |
|
Comment by James Muzerolle: Tested with 0.16.1, issues remain. Closing this and opening a new ticket (JP-1489). |
Issue JP-936 was created by James Muzerolle:
I am testing calwebb_spec3 using a set of simulated MOS exposures, which consists of a 3-shutter nod pattern. The level 3 products, with source-based files containing 2D unrectified spectra from all exposures and the combined 2D and 1D products, were generated correctly. However, the combined 2D products appear to have a couple of problems:
Data for the same source on both detectors are not being combined; the NRS2 data are being dropped entirely unless a particular source has no spectrum at all on NRS1.
The combined products tend to appear somewhat jumbled, and the source spectrum from the 3 nods don't seem to overlap (a screen shot of one example is attached). It's possible I have not included fully correct WCS pointing information in the headers since I wasn't clear on exactly what needs to go where (keyword values need to be added by hand in this case); I essentially cut-and-pasted the SCI headers from a "real" example generated from the nOPs test runs. Do I also need to make changes to the primary headers?
The input data, level 2b products, and the level 3 products for one source can be found in /grp/jwst/wit4/nirspec/Muzerolle/Files_spec3.
The text was updated successfully, but these errors were encountered: