Implement NIRCAM TSO grism extraction from CubeModel in extract 2D

[sosey]

In TSO mode, the data arrays are 3D, where each plane of the cube corresponds to the countrate image for a given integration within the exposure. I think we've already enabled the various calwebb_spec2 steps to handle this form of data (a "CubeModel", as opposed to a more normal 2D "ImageModel"). So the thing to ensure is that extract_2d does an extraction from the input 3D cube, producing an output 3D cube that is simply smaller in the first two (x, y) dimensions, but still has all of the axis 3 planes. The extract_1d step then knows to produce a 1D spectrum from each plane of the 3D cube that it's given.

[nden]

Just a reminder that the extracted arrays are now stored in a SlitModel, not ImageModel or CubeModel.

[hbushouse]

And I believe SlitModel is capable of doing 3D, right?

[nden]

Yes, SlitModel supports 2D and 3D arrays.

[hbushouse]

This is a "must have" for B7.2 in order to support level 4 requirement DMS-786.

[sosey]

@hbushouse if the team wants extract_2d implemented for TSO, I need to get the cross-dispersion size for the extraction for both full frame and subarray images since we don't have a measurement from the catalog to use. I can use crpix to get the size of the box in the dispersion direction.

[hbushouse]

As noted in #1235, the user is allowed to use a few different subarray sizes and full frame for NIRCam TSO grism exposures. All of the subarrays have their "bottom" edge located at the physical bottom edge of the detector and grow in size vertically. According to the TSO WG, the source spectrum trace will always be centered somewhere near row 34 in the vertical direction (dispersion running parallel to rows). So the larger subarrays will just result in larger amount of sky above the spectrum. The WG indicates that we can assume that TSO grism sources will always be point-like. They would like extract_2d to always produce a cutout that is 64 pixels in height (cross-dispersion direction) for all subarrays and full frame exposures, which is equal to the height of the smallest available subarray (2048 x 64). This is to allow area within the cutout for sampling the background in later steps, such as extract_1d.

Extract_1d, in turn, should use an extraction slit height of ~5 pixels within the 2D cutout, because the sources should be point-like. They would like the slit height to be a parameter that a user can set (during reprocessing) to tailor their results.

[philhodge]

I noticed from #1235 that the full-frame image has crpix2 = 485. If the spectrum will be near row 34 even for full-frame TSO grism data, will that value of crpix2 be a problem for extract_2d?

[hbushouse]

@philhodge Good catch. I'll ask the TSO group about that one.

[hbushouse]

Kevin Stevenson (TSO WG) agrees that the source will be located at CRPIX2=485 when full-frame readouts are used for NIRCam TS grism exposures. For consistency with the subarray modes, in terms of the spectral trace always ending up near row 34, they'd like to have extract_2d produce a 2D cutout that is slightly miscentered in the y direction so that the trace falls near row 34 in the cutout. Given that the 2D cutouts are 64 pixels high, this would mean centering the cutout near row 483 (CRPIX2 - 2).

[sosey]

quick question - I currently have this implemented to use the orders specified for extraction in the wavelengthrange reference file. (currently 1 and 2 for all filters). Users can also specify this as a parameter. Is there a team desire to only extract the first order though?

[hbushouse]

Good question. Let's get Kevin's input on this (I'll ask him). For TSO I'm guessing that they might only want order 1, as opposed to regular WFSS science where they could want both.

[stscicrawford]

cc: @kevin218 in case you can comment here

[sosey]

@NorPirzkal I must have asked you in person, I can't find the email at the moment, but for WFSS specifically, I also have both orders being extracted for all filters, is that still what the nircam team wants?

[npirzkal]

Yes, both orders should be identified and extracted. For two reasons: 1) the second order is present in only a subset of filters and small fraction of the FOV for these filters, so it does not add very much to the amount of data extracted, but 2) extracting the second orders is a very basic first step to understanding where these are and hence how they contaminate first orders.

[sosey]

@kevin218 @hbushouse how about the three of us try and get together in person to go over this. Any possibility to do this later this afternoon, or tomorrow some time?

[kevin218]

I can do 3-5pm today. I won't be around tomorrow.

[sosey]

Discussion results: nircam TSO team wants extract_2d to only make the cross-dispersion cut and keep the entire array in the dispersion direction. There will be no order extraction or anything else done during this step beyond making the cut and saving a SlitModel with the results. During extract_1d, the wavelengthrange file should be used to find where the first order begins and ends in the dispersion direction, add any padding that is needed to ensure we encapsulate the PSF, and then extract the flux in the cross-dispersion direction.

@philhodge is there anything else I could add in the background to extract_2d in order to help extract_1d?

[hbushouse]

If I remember correctly the discussion also resulted in the decision to only produce output for order 1 (is that right?). If so, will there be meta data of some kind passed along to extract_1d to let it know that it only needs to extract order 1?

[philhodge]

For NIRISS SOSS, extract_1d loops over the list [1, 2, 3] and extracts spectra with those spectral orders, if a matching entry is found in the reference file. If no matching entry is found for some spectral order, that order will be skipped. For other modes, the spectral order is assumed to have been assigned to slit.meta.wcsinfo.spectral_order by extract_2d. If slit.meta.wcsinfo doesn't exist, the spectral order defaults to 1.

[sosey]

@hbushouse nope. The discussion yielded that I wouldn't do anything to identify orders and just cut the cross-dispersion size to 64pixels. The team believes however that only order 1 will be present. But, extract_1d still will need to compute where the trace really falls in that direction to only pull the trace flux out and avoid adding extra noise from the surrounding pixels.

[sosey]

@philhodge yah, as it stands extract_1d will need to add the code that figures out where the first order is before extracting it. For NRC_TSGRISM they only want the first order pulled from the SlitModel that will come from extract_2d. I think that you should be able to just call the meta.wcs specifying the first order directly and get the position for the dispersion extents, you'll need to get the wavelengths from the wavelengthrange reference file for the filter used. I hesitate putting explicitly the order and wavelenth min,max in the wcsinfo right now because it's not strictly true for the data.

[philhodge]

@sosey By "and get the position for the dispersion extents" do you mean from meta.wcs.bounding_box? extract_1d does do that, but maybe not in all cases.

[sosey]

essentially. The bounding box for the grism is technically the whole image. For TSO there's only one object, always in a known location, so I've harded coded that source position, which really limits the valid row, but not the valid columns. I think that extract_1d wants to know the wavelengths at pixels and sum the flux, right? The wcs you should be getting will translate (x, y, order) <-> (wavelength, order). In order to find where to start extracting flux, I think you need to use the min and max wavelengths from the wavelengthrange reference file as input, then add padding that amounts to ~3 * the half-width half-max of the psf in that filter. @kevin218 should be able to provide those numbers. The cross-dispersion extraction size will always be 5 pixels.

[philhodge]

The center row (32?) and the extraction width 5 should be specified in the reference file.
extract_1d does not currently use the wavelengthrange reference file, but it could. One thing I'm concerned about is that if the zero-order image also falls on the detector, using the range of wavelengths to determine the relevant range of pixels in the dispersion direction could be ambiguous.

[sosey]

I can only go by what the team has specified.... but, since we are outputting a SlitModel I can set the values in the meta, for xstart, xsize, ystart, ysize as well as the wcsinfo for waverange_start, waverange_end, spectral_order, to be explicit for order 1, then hard code that to the WCS object to make sure it's the only one ever used. This negates anyone using the WCS object to calculate any other orders for the data coming out of extract_2d. This would get around extract_1d needing to use the reference file. It would then just extract pixels contained in the xsize,ysize box. If we don't do the above, and you want to add the cross-dispersion extraction size to the wavelengthrange reference file we need to add something there to hold that information just for TSO grism mode. The center row is set by the CRPIX values.

[sosey]

After discussion with phil: For now, since the extract_1d step doesn't take in the wavelengthrange reference file, I'm going to add the specific order 1 information to the WCS object and the slit/wcsinfo metadata of the output. extract_1d for this mode won't be called until sometime in the tso_3 pipeline.

[hbushouse]

extract_1d for this mode gets called twice: once at the end of calwebb_spec2 and again in the middle of calwebb_tso3 (after outlier_detection has been applied to the 2-D cutouts created upstream by extract_2d during calwebb_spec2). But it doesn't really matter how often or where it's called. The only important thing is that it's called after extract_2d and hence if extract_2d populates the kind of info described above in the wcs object and slit meta data, it should have what it needs. Except for the size of the aperture to use in the cross-dispersion direction. In the case of regular WFSS, it doesn't need that info, because it's just collapsing all the pixels in the cross-dispersion direction, due to the fact that the 2-D cutouts from extract_2d are already trimmed down to include only the signal from the source. But in the case of NRC_TSGRISM extract_2d will now be including a lot of extra real estate in the cross-dispersion direction, so extract_1d needs to be told (somehow) to use an aperture that's 5 pixels high in the cross-dispersion direction. How do we pass that along? For other spectral modes, it gets that info from the extract1d reference file, but it's assumed that for grism modes that reference file isn't needed. Can we create an extract1d reference file for NIRCam that only has an entry for the NRC_TSGRISM mode? I'd rather do that than hardwire the aperture size into the extract_1d code.

[sosey]

I thought we decided extract_1d wasn't called in calwebb_spec2? Either way, do you still want this submitted today for 0.10 @hbushouse? I need to know what I should submit today, and since extract_1d isn't currently implemented for TSO grism at all, we could just make the simple cutout, then have a meeting to discuss the best design going forward and make the resulting update in the few couple weeks.

[philhodge]

I'm pretty sure that if bounding_box is defined, extract_1d will trim the extraction region to only include what's in the bounding box. That could be used to center the extraction on row 34 and limit the extraction width to 5 pixels.

[sosey]

yah, nadia and I discussed that. To set the bounding box I still need to read the wavelengthrange reference file and calculate where the trace is (I would still need numbers from Kevin on the padding) and if it needs to go in today, I'd need to use the old wavelengthrange format to get that information...and then change it when we deliver the new schema and datamodel. I think my preference for a delivery today is just make the cutout and stop there. Then worry about the optimal way to get the information to extract 1d after..

[philhodge]

The bounding box doesn't have to set limits in the dispersion direction. For the time being, you could just set the cross-dispersion limits and leave the width to be the full width of the cutout.

[hbushouse]

This doesn't need to get in today to be included in 0.10.0, especially since the new wavelengthrange ref files aren't installed yet and extract_1d likely won't be able to handle the results anyway. So there's no rush.

By the way @sosey, extract_1d gets called at the end of calwebb_spec2 processing for any and all spectral modes. But the resulting exposure-level _x1d product is a "dead end" product, that's intended for quick-look only and is not passed along for any further processing. It's the _cal product from calwebb_spec2 that gets passed along as input to calwebb_tso3, where the _cal product contains the 3-D stack of 2-D cutouts created by extract_2d. So whether or not we apply extract_1d at the end of calwebb_spec2 for the TSGRISM mode, it doesn't affect the rest of the flow into level-3 processing.

[sosey]

okay. Can we remove the 0.10 milestone then? Thanks for the clarification on extract_1d.

[hbushouse]

hbushouse modified the milestones: 0.10.0, 0.11.0 20 hours ago