JP-3569: Update variance handling in resample step

[melanieclarke]

Resolves JP-3569

Closes #8344

Variance arrays from input data are now resampled individually, then combined with weights that match the science weight type, instead of being self-weighted. Input variance values equal to zero are propagated as zeros, rather than as NaN. In the final combined error array, values for which all input error components are NaN are set to NaN instead of zero. NaNs are otherwise ignored in the sum over variance components.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.42%. Comparing base (6580914) to head (835da72).
Report is 4 commits behind head on master.

❗ Current head 835da72 differs from pull request most recent head 0159e5f. Consider uploading reports for the commit 0159e5f to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8437      +/-   ##
==========================================
+ Coverage   56.38%   56.42%   +0.04%     
==========================================
  Files         387      387              
  Lines       38716    38754      +38     
==========================================
+ Hits        21830    21868      +38     
  Misses      16886    16886              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hbushouse]

@melanieclarke Pro tip: If you include the name of the JP ticket in the PR title, it'll automatically create links between the two. e.g. "JP-3569: Update variance ..."

[melanieclarke]

I think this is ready for review now, if someone can start a regression test run for me.

[drlaw1558]

LG2M from the science side; thanks for adding the doc update.

[melanieclarke]

Regression tests started here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1413/

[melanieclarke]

Reviewing the regression tests, it looks like all changes are to post-resampling products (i2d, s2d, x1d, or catalog), in the error or variance arrays only as expected:

• In the ERR array in s2d and i2d files, there are now NaNs where there used to be 0s around the edges of the images. This is because I added some handling to catch when all input components are NaN and return NaN instead of 0 when they are added with np.nansum.
• In the variance arrays in s2d and i2d images, away from the edges, there are now 0s where there used to be NaNs. This is due to the change in weighting in the variance combination: since inverse variances are no longer used in the propagation, 0 is an allowed value.
• In the error arrays in the x1d images, there are now values where there used to be NaNs. This is due to the above change allowing 0 values.
• Most variance and error values are a little smaller, due to the change in the drizzling method for the variance components.

One test result I don't understand and will look into a little closer:

• The catalog output error columns contain some new NaNs where they used to have values, for both MIRI and NIRCam.

The NIRISS filter rotation failure in assign_wcs looks unrelated.

[melanieclarke]

It looks like the catalog NaNs correspond to sources with saturated cores, for which the variance components are all NaN in the i2d file. The ERR is now set to NaN at these locations, instead of 0. The aperture photometry in the source_catalog does not ignore NaNs, which is why NaN values now appear in the error columns in the output catalog.

They do not also appear in the flux columns because fillval=INDEF in resample, by default, so missing values are set to zero in the SCI data. If the fillval default is set to NaN, saturated stars will also report flux = NaN.

@hbushouse - Should I add a mask for NaN values to the source_catalog aperture photometry, leave it as is, or file another ticket?

[melanieclarke]

Per @drlaw1558 and the JP coordination meeting today - the issue with NaN values in aperture photometry needs a separate ticket. I will leave the code as is in the catalog step.

[hbushouse]

Looks good to me, including test results and updated docs.