TSO processing in calwebb_spec2

[karllark]

The TSO observations need to have slightly different processing in calwebb_spec2 than the rest of the observations.
Specifically the only steps that are done are WCS info, flat field correction, flux calibration, and rectified 2D/3D product.

[jdavies-st]

To clarify, does this apply for both MRS and LRS data?

[karllark]

This applies to all spectroscopic TSO data regardless of instrument.

[hbushouse]

Right now the only MIRI spectroscopic mode that is treated as TSO by default is MIR_LRS-SLITLESS. Eventually the TSO option will be available on all modes, so that could then include both MIR_MRS and MIR_LRS-FIXEDSLIT (as well as MIR_IMAGE, but that's a calwebb_image2 issue).

Handling IFU data, which is already in 3-D form, as TSO, which is done on a per integration basis, will be a MAJOR overhaul of the entire pipeline, because now we're talking about having to handle 4-D data structures.

[hbushouse]

At the request of the NIRSpec team, the NIRSpec BRIGHTOBJ mode, which is a TSO mode that acquires data through the S1600A1 fixed-slit, also includes the extract_2d in calwebb_spec2 processing, just as non-TSO S1600A1 do, because the defined region of the spectral trace does not fill the entire subarray used to acquire the data.

[hbushouse]

TSO exposures do NOT produce a rectified 2D/3D product, either in calwebb_spec2 or calwebb_tso3. The desire is to do as little harm as possible to the data. 1-D spectral extractions are done on the unrectified/unresampled 2-D data.

[karllark]

I am confused. All spectroscopic data should produce 2D/3D rectified products. This is a product for the archive, not for use later in the pipeline. So, there should be no reason why TSO exposures are any different than non-TSO exposures.

[karllark]

Is the NIRISS SOSS mode not a TSO by default mode? This has been my understanding.

[karllark]

Check out the last entry in the table on this JDox page. This is my understanding of the modes that have TSO options. Note that there is no TSO option for MIRI IFU (MRS) observations at this point.
https://jwst-docs.stsci.edu/display/JDAT/By+Observing+Template

[karllark]

Good point about the extract_2d step. Makes sense and I've updated the confluence page to reflect this.

[hbushouse]

As of right now the modes that are treated as TSO by default (because we don't yet have a specific TSO flag from APT) are NIRCam TS image (NRC_TSIMAGE), NIRCam TS grism (NRC_TSGRISM), NIRSpec BrightObj (NRS_BRIGHTOBJ), NIRISS SOSS (NIS_SOSS), and MIRI LRS slitless (MIR_LRS-SLITLESS).

[karllark]

Sounds reasonable. Any ETA on getting the TSO flag from APT through the system?

[hbushouse]

I don't recall right now which APT or PPS release this is supposed to be in. Will need to check. We have already implemented a TSOVISIT keyword in the pipeline systems that will get populated by this APT/PPS flag, but so far we haven't seen it populated in the PPS DB tables for any test data.

[stscieisenhamer]

FYI: The overall issue is PR 87867

[hbushouse]

PR 87867 indicates, in a spreadsheet, the following for each exposure type:

EXP_TYPE	TSO status
MIR_IMAGE	Allowed; default=no
MIR_LRS-FIXEDSLIT	Not allowed
MIR_LRS-SLITLESS	Allowed, default=yes
MIR_MRS	Not allowed
MIR_LYOT, MIR_4QPM	Not allowed
NRC_IMAGE	Not allowed
NRC_CORON	Not allowed
NRC_TSIMAGE	Always
NRC_TSGRISM	Always
NRC_GRISM	Not allowed
NRS_FIXEDSLIT	Not allowed
NRS_IFU	Not allowed
NRS_MSASPEC	Not allowed
NRS_BRIGHTOBJ	Always
NIS_IMAGE	Not allowed
NIS_WFSS	Not allowed
NIS_SOSS	Allowed, default=yes
NIS_AMI	Not allowed

So this indicates that the only modes that can be toggled are MIR_IMAGE, MIR_LRS-SLITLESS, and NIS_SOSS. Right now we treat MIR_IMAGE as non-TSO, and the remaining two as TSO by default. So logic will need to be added to all of the applicable pipeline modules, and ASN generator rules, to allow for toggling of those 3 modes.

[stscieisenhamer]

Which has been implemented. The current logic is to create TSO associations when either an EXP_TYPE is one of the "always TSO" or the IS_TSO flag is set.

[stscieisenhamer]

Question

Architecture question: Is this to be baked into the calwebb_spec2 logic, or can we rely on just a different configuration file?

[hbushouse]

For level-2a processing we've gone the route of using a different configuration file, calwebb_tso1.cfg. The consensus was that we didn't want to have it hardcoded into the pipeline module itself, because that would prevent users from making their own choices as to what they wanted to apply. So presumably we will now end up with an entire series of TSO configuration files, but we need to figure out appropriate unique names for the ones that govern calwebb_image2 and calwebb_spec2, because we can't call them both calwebb_tso2.

[sosey]

can we do calwebb_tso_spec2 and calwebb_tso_image2 ?

[hbushouse]

That's a reasonable possibility. But then for consistency should we go back and rename calwebb_tso1 to calwebb_tso_detector1? Of course once we get to level-3 processing the TSO pipeline is setup to handle both image and spec, so we're back to just calwebb_tso3, instead of calwebb_tso_image3 and calwebb_tso_spec3.

[stscieisenhamer]

Proposal on full specification: calwebb_<mode><stage#> where:

• stage#: 1, 2, 3
• mode:
  • detector: any detector
  • spec: any spectral
  • image: any image
  • tso: any tso
  • tso_image: image-based tso
  • tso_spec: spectral-based tso
  • ami: any ami (yes there is only one but...)
  • etc. etc. etc.

Renaming calwebb_tso1 to calwebb_tso_detector1 is slightly redundant. The issue is actually the use of detector. One could say that the most generic of the pipelines, which take nearly any input, should be called level or stage. So calwebb_detector1 should be calwebb_stage1. But since this is the only case, IMHO, we could let that one go.

And yes, already on coffee #2...

[stscieisenhamer]

Implementation

A configuration, calwebb_tso_spec2.cfg, will be created only allowing the following steps:

• assign_wcs
• extract_2d
• flat_field
• photom
• extract_1d

To Do

• documentation
• tests
• code
  • setup calwebb_tso_spec2.cfg.
  • Create Level2 rules to create a new association type, tso_spec2, associations to indicate that the new config is to be used.
  • File SDP issue for the new configuration & association types.
• tests
• documentation

[stscieisenhamer]

Question

So, do to a bunch of already-baked in constraints, the only extra steps that would be executed is background and source type using the standard config. Source type is a non-modifyier, so I'm guessing whether its executed or not is not an issue.

For MIR_LRS-SLITLESS, APT does allow off-target background. So, is background alright to keep on the list? In general, backgrounds won't be available unless APT allows it. If allowed, I am presuming it should be executed.

If doing background is OK, there is nothing to be done, unless I'm missing something.

[stscieisenhamer]

@hbushouse @sosey Any answer to the above question?

[hbushouse]

Regardless of the answer to the question about background, the pathloss steps also needs consideration. NIRSpec BrightObj, which is a TSO mode, is obtained using the S1600 fixed-slit. NIRSpec fixed-slit data normally have the pathloss step applied, so in the case of BrightObj it would need to be skipped. So given that we will need a custom cfg to at least skip the pathloss step, then we might as well include skipping the background too.

[hbushouse]

Oh, and pathloss correction is also about to be implemented for NIRISS SOSS mode too, but given that SOSS mode is TSO by default, application of pathloss is in conflict with the JCCWG table specifying which steps to apply/skip for TSO. Clearly we need further discussion with the JCCWG and TSO WG.

[hbushouse]

To summarize:

• calwebb_image2: The only step marked by the JCCWG to be skipped is background, but we also need to skip resample
• calwebb_spec2: The steps marked by the JCCWG to be skipped are background, source_type, and pathloss, but we also need to skip resample. They also list skipped steps of imprint, msaflagging, straylight, fringe, and barshadow, but as of right now the only modes that allow for TSO do not get any of these steps applied anyway (they're all NIRSpec IFU/MOS and MIRI MRS steps).

We need to skip resample for spectral data, because we simply can't do resampling for some modes, like NIRISS SOSS and MIRI LRS, due to the lack of complete WCS transforms that are needed for resampling. Also, given the fact that the level-2b resampled products are just "dead end" products intended for quick-look and are not passed on for further processing, there is no real need to waste the processing time and effort to produce them, even for the modes where it's possible to do so (e.g. imaging).

Because we need to skip 1 or more steps in both calwebb_image2 and calwebb_spec2, we will need custom cfg's for both and hence we'll need to adopt the naming convention proposed above, e.g.
calwebb_tso_image2.cfg
calwebb_tso_spec2.cfg

[sosey]

@hbushouse so no need to implement background for NRC_TSGRISM, right? (thinking of #1710)

[hbushouse]

Correct, no background step in calwebb_spec2 for TSO modes.

[stscieisenhamer]

Unblocking and going with the separate config files, as specified above.

[karllark]

Looks good!