NIRSPEC: MOS Flat field #5328
Comments
|
Comment by Jane Morrison on JIRA: The following information is from JP-1663 (which was closed with PR 5284) but there are still remaining differences between the NIRSPEC and JWST flat field step for NIRSpec MOS data. Hi Jane Morrison, I am using jwst pipeline version: 0.17.1.dev4+g944cd605, and I made all the changes you requested to the NIRSpec code, you can find the new version at: I placed the new files at: ████████████████████████████████████████████████████████████████████████ Same convention:
The differences we are getting are still too large, something else is not quite the same in both calculations of the flat so we need to keep investigating.
The difference in the flat filed between NIRSPEc and JWST pipeline is: !image-2020-09-14-09-10-52-328.png|width=719,height=234!
The histogram of the NIRSPEC interpolated flat, JWST interpolated flat and difference of these flats is: |
|
Comment by Jane Morrison on JIRA: Copied comment from 1663 (which is closed fixing some of the NIRSPEC flat field issue except for this remaining one) |
|
Comment by Jane Morrison on JIRA: I finally got back to this issue. In the jwst pipeline I set the d_flat and f_flat to 1. I also set the corresponding dq flags for these flats to 0. I ran the pipeline and produced the following output. I have displayed the JWST results on the top and the NIRSPEC pipeline on the bottom. This is for the short wavelength end. I am confused about why the NIRSpec output has an sflat with values other than 1 that is only 2 pixels high. Hum any ideas ? I will go look at the NIRSpec pipeline and see if I can untangle this. I will also try not setting the f and d DQ flat values to 0. To see if the makes any difference. !image-2020-10-16-13-24-29-627.png|width=859,height=406!
|
|
Comment by Jane Morrison on JIRA: I removed setting the f and d flat DQ values to 0 and it did not make a difference in the JWST product. So it our differences are how the s flat is interpolated. More after I dig into both pipelines
|
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero James Muzerolle I have downloaded (cloned) the NIRSPEC pipeline. I thought I might try and run it. But first I have a question. In the nirspec pipeline I have a question about lines 459 and 461. kk = np.size(zzwherenonan). I think the slat for a particular wavelength range is only interpolated if kk = 2. Why does k need to be 2 ? I am wondering if this is why the nirspec sflat is very skinny ? |
|
Comment by James Muzerolle on JIRA: Jane Morrison That is a check to ensure that there are valid correction values bracketing the wavelength value of a science pixel, so that an interpolation can be done. There are checks both for NaNs and DQ flags. Last month, we did find a mistake in the DQ flag conditional (line 465); it should only reject something with DQ=1. It looks like that bug has not yet been fixed; Maria Pena-Guerrero, can you take a look at that? |
|
Comment by Jane Morrison on JIRA: James Muzerolle Maria Pena-Guerrero Ah I see that now. That makes sense with what I am seeing in the interpolated s flat from the nirspec pipeline. Much of the data = 1 because the DQ flags may not be 0. |
|
Comment by Maria Pena-Guerrero on JIRA: ok, so I have changed the code of lines 465-467 (not pushed yet), it now says this: so if the DQ value at location pind is 1 then sfs=1.0, and similarly in the f-flat (lines 486-488).
|
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero I defer to what you think should be done. But does the size of kk have to equal 1, is it just have to be 1 or greater ?
|
|
Comment by Maria Pena-Guerrero on JIRA: yes, you are correct. Thanks for catching this bug Jane Morrison! I changed it. |
|
Comment by James Muzerolle on JIRA: Maria Pena-Guerrero I think the second statement needs to be " if np.size(kk) != 0:". We only want to set the correction value to 1 if one or both of the bracketing reference file pixels have DQ=1. |
|
Comment by Maria Pena-Guerrero on JIRA: James Muzerolle, I think those 2 statements are the same, I mean " if np.size(kk) != 0:" and " if np.size(kk) >=1:" in this case imply the same. It is the previous statement that looks if there are any DQ=1 values in the bracketing reference file pixels. |
|
Comment by James Muzerolle on JIRA: Oops, you're right, sorry. I misread Jane's comment as "greater than 1". |
|
Maria when you get a chance could you re-run the nirspec pipeline on the dataset we are testing and I will run my branch of the jwst pipeline on it (that sets the d and f flat to 1) and then we can see where we are on remaining differences. |
|
Comment by Maria Pena-Guerrero on JIRA: Sure, already running it. I'll let you know when its done. |
|
Comment by Maria Pena-Guerrero on JIRA: Jane Morrison, I put the new outputs at ██████████████████████████████████████████████████████████████████████████████████████████████████████ |
|
Comment by Jane Morrison on JIRA: Something does not seem correct. We used to have a difference close to 0 in the center of the slit. Now we do not have that any more. The difference is much larger (~0.2). Maria I want to make sure we are starting with the same input file. Could you point me to the input extract2d file that you used for this test. I want to first double check we are using the same one.
|
|
Comment by Jane Morrison on JIRA: Oh I see now. Maria Pena-Guerrero please confirm that the NIRSPEC pipeline you can does not set the D flat and F flat to 1.
|
|
Comment by Jane Morrison on JIRA: Here is another way of looking at the first slit - this is an image of the difference between the two pipelines for the first slit. The top plot is for the short wavelength region and the bottom plot is for the longer wavelength. There is a bit of overlap between the two images. There might be a few things we could continue to look at - like the bluish region in the top plot. Values there are ~ -0.01, but just a few. Maria Pena-Guerrero I think my branch for this result is the same and what is on master (I just have added some extra print statements). So you could run your comparison test. In the directory that you pointed me at there was a file: g140m_line1_NRS1_flat_comp.fits. I thought that was the comparison between the two pipelines but it seems to only contain 999 values. So maybe that is not what I thought. !image-2020-10-23-16-55-16-480.png|width=687,height=235!
|
|
Comment by Maria Pena-Guerrero on JIRA: Jane Morrison, I re-ran my test against the pipeline dev version just now and I placed the results at: ██████████████████████████████████████████████████████████████████████████████████████████████████████ I erased everything in there before becauseI think I used an old version of the pipeline and I was not sure if I had forced the d- and f-flats to 1. In these results, I am sure that the d- and f-flats to 1. I see differences of the order of 10^-5 and 10^-6, which is much better than before. Is this what you see too?
|
|
Comment by James Muzerolle on JIRA: Maria Pena-Guerrero Residuals of that order are in the ballpark of where we want to be, I think. Can you post some example histograms and standard deviations? I just want to make sure there aren't many significant outliers. |
|
Comment by Jane Morrison on JIRA: Here are the differences for the first 6 slits. Maria Pena-Guerrero Close enough ? Or should we investigate some of the larger difference !image-2020-10-28-17-18-35-108.png|width=749,height=749!!image-2020-10-28-17-19-17-088.png|width=731,height=731!
|
|
Comment by James Muzerolle on JIRA: That definitely looks much better! There are quite a few significant outliers, though; are those from pixels on the edges of the traces? Or possibly other differences in how flagged pixels are being treated? Can you show the difference images? |
|
Comment by Jane Morrison on JIRA: James Muzerolle So the plot I displayed oct 23 is the difference image of the first slit. I put the NIRSpec interpolate flat, JWST interpolate flat and difference image for the first 6 slits at ███████████████████████████████████████ You can look at those products and see what you think. I think some of the difference is where the dflat comes into play. |
|
Comment by James Muzerolle on JIRA: Thanks, Jane Morrison. Looking at the first two of those slits, I see some strange patterns. There seem to be repeated segments of about the same length containing clusters of pixels with large positive and negative differences. Here's one example, with the stretch increased to better show the deviations: !Screen Shot 2020-10-29 at 6.27.43 PM.png|thumbnail! Were both versions run with all dflat and fflat set to 1? If so, these outliers are related to something in the sflat. |
|
Comment by Jane Morrison on JIRA: THe D and F flat are not set to 1 in this run. I originally thought was the plan but when I did that I had large differences in the two pipeline results. I went back and removed the setting the D and F flat to 1 and got MUCH closer results. Maria Pena-Guerrero can you confirm that the NIRSPEC pipeline is using the D and F flats ? (not setting them to 1).
|
|
Comment by Maria Pena-Guerrero on JIRA: Jane Morrison, for the files I recently placed in central store both D and F flats are indeed set to 1 in the NIRSpec pipeline. I am running now a test where they are not forced to 1. |
|
Comment by Jane Morrison on JIRA: Ok we might have a bigger problem then. Lets see how your results compare with setting D and F to 1 and not seeing it to 1. There was a big difference when I did that. I will create an interpolated flat field setting D and F to 1. |
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero James Muzerolle Ok I think we need to get together on web-ex or something to discuss these results. I have compared the JWST pipeline results with NIRSpec pipeline both for setting the d and f flat = 1 and not setting it to 1 but using it. I have several questions for you guys - probably best done in where we can talk to one another Here are the results !image-2020-11-06-10-09-01-820.png|width=833,height=469!
|
|
Comment by Maria Pena-Guerrero on JIRA: Jane Morrison, here are the files for the d- and f-flats = 1: And here are the files for the s- and f-flats = 1: hope this is helpful |
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero James Muzerolle First result from test. If you guys want the jwst interpolated flat from the run of setting D and F flat = 1 some place let me know. I tried to write it to Maria's directory but I did not have permission. !image-2020-11-10-13-16-33-128.png|width=833,height=369! |
|
Comment by Jane Morrison on JIRA: James Muzerolle Maria Pena-Guerrero I still do not know why the S flat seems different between the pipelines. I am letting you guys research that and see if it makes sense (plot above) I have moved on to looking at setting S and F flat to 1. I don't understand these results either. The top plot (below) is the interpolated D flat only from the jwst pipeline. The middle plot from the nirspec pipeline. This is for the first slit in the data. I went and grabbed the full D Flat : jwst_nirspec_dflat_0001.fits This region is y: 1315 to 1329 and x: 1542 to 2048. When I look at this region in ds9 all the values are around 1 for the 39 planes. I am not totally sure how the fast variation would change this but I don't think it would result in values that are shown on the NIRSPEC plot > 2.5. |
|
Comment by James Muzerolle on JIRA: Sorry for the delayed reply, Jane Morrison, I've been knee-deep in proposal work. I don't understand the differences in the S flat. The purple region in our version may be a flag issue? One thing we should check is that we're using the same wavelengths, since the correction values are wavelength-dependent. Looking at the fast vector for the D-flat, the correction factors over the wavelengths of G140M should range from ~0.9 to 1.3. So values > 2 don't make any sense. Maria Pena-Guerrero can you check that? Maybe the S-flat wasn't actually turned off? |
|
Comment by Maria Pena-Guerrero on JIRA: James Muzerolle could there be any difference in the reference files we use from CDP3 that lives in central store compared to what is in CRDS? I tried running our code on the CRDS files but since the structure is different, it is not so straightforward. |
|
Comment by Maria Pena-Guerrero on JIRA: I realized that if I force the f-flat to 1, as well as the s- or d-flat, I get different results. I was not doing this before because I thought the reference file had all 1.0 values |
|
Comment by James Muzerolle on JIRA: The contents of the original reference files on central store should be the same, just a different format. Are you saying you get different results if you set the f-flat to 1, but keep the other components the same? That is odd, because the f-flat reference files should have values of 1 for all wavelengths/apertures. Could there be some sort of edge effect from interpolation, or are the correction values for interior pixels also different? |
|
Comment by Maria Pena-Guerrero on JIRA: I see that we have a bunch of wavelengths > 1.84, and I don't understand why. From the file header: |
|
Comment by James Muzerolle on JIRA: Maria Pena-Guerrero The current version of the spectral configurations file has the max wavelength range for G140M+F100LP set to 0.97-1.89. Do you see wavelengths larger than that? Is the pipeline truncating at 1.84? |
|
Comment by Maria Pena-Guerrero on JIRA: Oh ok, I see values like 1.895575226367663 but not greater than 1.896. And also, duh me! I was making the changes on a script and running the wrong script, so yes, the s- and f- flats where not "off" in the results. I am running the right script now. |
|
Comment by Maria Pena-Guerrero on JIRA: ok, Jane Morrison, sorry for all this mixup on my part. Here are the correct files: █████████████████████████████████████████████████████████████████████████████████████████████████████ █████████████████████████████████████████████████████████████████████████████████████████████████████████ |
|
Comment by Jane Morrison on JIRA: @maria James Muzerolle Maybe I am doing something wrong. But I grabbed the setting S and F to 1 and I get exactly the same results. There are many values in the nirspec pipeline with D values > 2.5. Maria can you check it you also find that.
|
|
Comment by Maria Pena-Guerrero on JIRA: Jane Morrison, I do not see any values > 1.3 for the D flat. I just added the _calc files to this ticket to make it easier. They are called g140m_line1_NRS1__flat_calc_Dflat.fits and g140m_line1_NRS1__flat_calc_Sflat.fits |
|
So is g140m_line1_NRS1__flat_calc_Dflat.fits the result from the NIRSPEC pipeline for the interpolated flat when S and F are = 1 |
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero James Muzerolle Ok I must of grabbed the wrong data. I blew it away and started fresh. Here the result for S = 1, F =1, so only the D flat is used. Looks good me. How about you guys ? I looked at at few slits and found similar results. So I will look at the other case. !image-2020-11-20-12-40-58-253.png|width=703,height=551! |
|
Comment by Jane Morrison on JIRA: Maria Pena-Guerrero James Muzerolle Here are the results for comparing the first slit with D=1 and F=1. First thing I want to sort out is the large number of 1 values in the NIRSPEC result (middle) . I am not sure how the NIRSpec pipeline is set up. Is it possible that setting D = 1 and F =1 could be overriding the S flat value ? The jwst pipeline D = 1 F =1 Then interpolated flat = flat_2d = f_flat * s_flat * d_flat !image-2020-11-20-13-00-32-163.png|width=724,height=565!
Below is the difference image for the first slit. The edges of the slit seem to be the problem: blue on left and orange on right. James Muzerolle does this difference make sense and do you have an idea of where it is coming from ? !image-2020-11-20-13-06-01-570.png|width=711,height=189! |
|
Comment by Maria Pena-Guerrero on JIRA: I think this is because I did not comment the line if ffs == 1.0 and sfs == 1.0 and dfs == 1.0: I just did another run with these 2 lines commented out. I am attaching this new file, it is called g140m_line1_NRS1__flat_calc_Dflat2.fits |
|
Comment by Jane Morrison on JIRA: I may not of been clear. I think that when we both set S and F =1 the results are in agreement. See comment on Nov 20 at 2:40 pm. So that means the way we handle the D flat seems consistent. What is still a problem is when the D and F =1 so we are only looking at the S flat. Our S flat is different. See comment on Nov 20 3:06 pm.
I think your g140m_line1_NRS1__flat_calc_Dflat2.fits is when S and F =1 so we are only looking at the D flat. I want to see just the S flat - so when D and F = 1. Does that make sense ? I am not too sure that commenting out that line if ffs == 1.0 and sfs == 1.0 and dfs == 1.0, will make any difference. I think it is something else.
|
|
Comment by James Muzerolle on JIRA: Maria Pena-Guerrero Jane Morrison I've finally had a chance to take a look at this, sorry for the delay. I did a quick check of our results for the above slit; it looks like the pixels with a value of 1 are all located in one of two wavelength ranges: lam < 0.975 and lam > 1.25. 0.975 corresponds to the wavelength of the first "slice" of the Sflat MOS cube. Correction values should not be extrapolated, so the expected behavior for that component should be that the factor is set to 1. However, the other component, the "fast" vector, does have legitimate reference values at those wavelengths, so the corrections should be calculable in that case, and hence the combined Sflat values (interpolated cube value * interpolated fast value) should not be 1. I checked several other slits, this seems to occur in every case. I'd guess our code is not doing that combination correctly? I haven't had a chance to look at the code, so Maria should verify. For the other extreme (lam>1.25), I'm not sure why the values are being set to 1, since there are valid bracketing reference values in both the cube and fast vector. Of the 5 slits I checked, I saw this behavior in only 1 other, but at lam > 1.85. ? |
|
Comment by Jane Morrison on JIRA: We just started a new sprint. If we can get this problem sorted out (if there is an error in the jwst pipeline) we can get this in build 7.7 if we get it done this sprint. So let me know how I can help with this issue
|
|
Comment by Maria Pena-Guerrero on JIRA: Hi Jane Morrison, can you please put your files with forced values to 1 somewhere? (I mean to separately investigate the s- and d-flats) Thanks! |
|
@penaguerrero I put the results on |
|
Comment by Maria Pena-Guerrero on JIRA: Hi Jane Morrison! thank you so much for the data and the code. I was able to run a comparison and found a couple of bugs in our code. So I can now happily report that the test is PASSING and that the differences between the pipeline and our code are right in the range where we want them to be. |
|
That is great news. I will close ticket JP-1691. |
|
Comment by Jane Morrison on JIRA: The differences between the NIRSpec and JWST pipeline for MOS flat fielding have been resolved. The NIRSpec team found some bugs in their pipeline. Now the JWST and NIRSpec results for MOS flat fielding agree |
|
Is this the PR that fixed this issue for the NIRSpec testing tool? spacetelescope/nirspec_pipe_testing_tool#170 I just want to close the loop. |
|
Yes I believe it is |
Issue JP-1691 was created on JIRA by Jane Morrison:
This ticket is the remaining work not finished in JP-1663. PR 5284 was merged in the JWST Pipeline and fixed some of the NIRSPEC MOS issues but there are remaining differences between the NIRSPEC and JWST flat field pipeline steps
The text was updated successfully, but these errors were encountered: