JP-3550: Multi-integration jump detection (#242)

* changes for charge spilling from saturated core * Update jump.py * Update jump.py * Update jump.py * Update saturation.py * Update saturation.py * use ellipses for all jumps * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * fix sat expand * fix inside ellipse test * Update jump.py * Update jump.py * Update jump.py * tests etc * return new dgq for sat expend * fix super expanding ellipses * Update jump.py * Update jump.py * wrong shape for edge too many expands due to edge being wrong * update edge parameter * remore print * Update saturation.py * fix miri showers * updates * Update twopoint_difference.py * Update twopoint_difference.py * retro fit sigman_clip * Update twopoint_difference.py * Update twopoint_difference.py * updates for test * Update twopoint_difference.py * fix multint * tests * Update twopoint_difference.py * fixed axis=1 to axis=0,1 * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py print statements * fix flagging fix flagging * Update jump.py * Update twopoint_difference.py * type problem * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * test sigma clip * Update twopoint_difference.py * Update jump.py * Update twopoint_difference.py * Update twopoint_difference.py * add total CR count * Update twopoint_difference.py * Update jump.py * Update jump.py * Update twopoint_difference.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update jump.py * Update twopoint_difference.py * Update jump.py * Update jump.py * Update jump.py * Update twopoint_difference.py * Update twopoint_difference.py * tests * Update jump.py * lower sig clip thres * Update twopoint_difference.py * Update twopoint_difference.py * Update jump.py * Update twopoint_difference.py * Update twopoint_difference.py * group diffs * Update jump.py * Update jump.py * Update jump.py * Update test_twopoint_difference.py * Update twopoint_difference.py * updates * updates * new test * Update twopoint_difference.py * Update jump.py * Update jump.py * change to nints for min groups * Update test_twopoint_difference.py * fixes * Update twopoint_difference.py * Update twopoint_difference.py * update * Update twopoint_difference.py * update * Update twopoint_difference.py * Update twopoint_difference.py * update * Update twopoint_difference.py * cleanup * Update jump.py * remove fits * add stddev output * updates * test updates * Reduce log output * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update twopoint_difference.py * Update jump.py * Update test_twopoint_difference.py * Add sigma clip to shower detection * remove or mark as skip extraneous tests * remove ds_store * remove DS_Store * Delete .DS_Store * Delete .DS_Store * Update CHANGES.rst * unpacked the use of shape to be named variables * replacing shape with meaningful variable names * fix style issues * Update test_twopoint_difference.py * Delete setup.cfg * Fix missing loop in flag 4 neighbors I added a loop over integrations that was missing due to the post CR flagging being done once for all ints. * Update twopoint_difference.py Remove loop over integrations and use where to get integration. * Update CHANGES.rst * Update CHANGES.rst * Update src/stcal/jump/jump.py * Update src/stcal/jump/jump.py * Update twopoint_difference.py Resolve comments on PR. * Update twopoint_difference.py * Update twopoint_difference.py * updates * Update twopoint_difference.py * updates * update tests * updates * Update twopoint_difference.py * updates * update * Update twopoint_difference.py * Update twopoint_difference.py * updates * Update twopoint_difference.py * Update jump.py * Update jump.py * fix clipping * Update twopoint_difference.py * new output * unit test updates * Update test_twopoint_difference.py * Cleanup * Update test_twopoint_difference.py * Update CHANGES.rst * Update CHANGES.rst * update variable name and parameter docs * Update .gitignore * Update twopoint_difference.py * Update twopoint_difference.py * Update test_twopoint_difference.py * clean up flag after * Update test_twopoint_difference.py * fix merge problems * Update jump.py * add test from JWST * change threshold flagging back to scalar * make sure copies are made of model attributes * Update test_jump.py * Update test_jump.py * Update test_twopoint_difference.py * Update test_twopoint_difference.py * Update CHANGES.rst * addressed PR comments --------- Co-authored-by: Howard Bushouse <bushouse@stsci.edu>
spacetelescope · Mar 20, 2024 · def387c · def387c
1 parent 92be82d
commit def387c
Show file tree

Hide file tree

Showing 6 changed files with 640 additions and 367 deletions.
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -20,6 +20,14 @@ Bug Fixes
 Other
 -----
 
+jump
+~~~~
+
+- Enable the use of multiple integrations to find outliers. Also,
+  when the number of groups is above a threshold, use single pass
+  outlier flagging rather than the iterative flagging. [#242]
+
+
 1.6.1 (2024-02-29)
 ==================
 

diff --git a/src/stcal/jump/jump.py b/src/stcal/jump/jump.py
@@ -1,14 +1,14 @@
 import logging
 import multiprocessing
 import time
+
 import numpy as np
 import cv2 as cv
 import astropy.stats as stats
 
 from astropy.convolution import Ring2DKernel
 from astropy.convolution import convolve
 
-
 from . import constants
 from . import twopoint_difference as twopt
 
@@ -18,10 +18,10 @@
 
 def detect_jumps(
     frames_per_group,
-    data,
+    indata,
     gdq,
     pdq,
-    err,
+    inerr,
     gain_2d,
     readnoise_2d,
     rejection_thresh,
@@ -56,6 +56,7 @@ def detect_jumps(
     minimum_groups=3,
     minimum_sigclip_groups=100,
     only_use_ints=True,
+    min_diffs_single_pass=10,
     mask_persist_grps_next_int=True,
     persist_grps_flagged=25,
 ):
@@ -81,7 +82,7 @@ def detect_jumps(
     frames_per_group : int
         number of frames per group
 
-    data : float, 4D array
+    indata : float, 4D array
         science array
 
     gdq : int, 4D array
@@ -90,7 +91,7 @@ def detect_jumps(
     pdq : int, 2D array
         pixelg dq array
 
-    err : float, 4D array
+    inerr : float, 4D array
         error array
 
     gain_2d : float, 2D array
@@ -109,7 +110,7 @@ def detect_jumps(
     four_grp_thresh : float
         cosmic ray sigma rejection threshold for ramps having 4 groups
 
-    max_cores: str
+    max_cores : str
         Maximum number of cores to use for multiprocessing. Available choices
         are 'none' (which will create one process), 'quarter', 'half', 'all'
         (of available cpu cores).
@@ -124,11 +125,11 @@ def detect_jumps(
         neighbors (marginal detections). Any primary jump below this value will
         not have its neighbors flagged.
 
-    flag_4_neighbors: bool
+    flag_4_neighbors : bool
         if set to True (default is True), it will cause the four perpendicular
         neighbors of all detected jumps to also be flagged as a jump.
 
-    dqflags: dict
+    dqflags : dict
         A dictionary with at least the following keywords:
         DO_NOT_USE, SATURATED, JUMP_DET, NO_GAIN_VALUE, GOOD
 
@@ -209,6 +210,15 @@ def detect_jumps(
     min_sat_radius_extend : float
         The minimum radius of the saturated core of a snowball for the core to
         be extended
+    minimum_groups : int
+       The minimum number of groups for jump detection
+    minimum_sigclip_groups : int
+       The minimum number of groups required to use sigma clipping to find outliers.
+    only_use_ints : boolean
+       In sigma clipping, if True only differences between integrations are compared. If False,
+       then all differences are processed at once.
+    min_diffs_single_pass : int
+       The minimum number of groups to switch to flagging all outliers in a single pass.
 
     Returns
     -------
@@ -235,13 +245,12 @@ def detect_jumps(
 
     # Apply gain to the SCI, ERR, and readnoise arrays so they're in units
     # of electrons
-    data *= gain_2d
-    err *= gain_2d
+    data = indata * gain_2d
+    err = inerr * gain_2d
     readnoise_2d *= gain_2d
     # also apply to the after_jump thresholds
     after_jump_flag_e1 = after_jump_flag_dn1 * np.nanmedian(gain_2d)
     after_jump_flag_e2 = after_jump_flag_dn2 * np.nanmedian(gain_2d)
-
     # Apply the 2-point difference method as a first pass
     log.info("Executing two-point difference method")
     start = time.time()
@@ -280,6 +289,7 @@ def detect_jumps(
             minimum_groups=3,
             minimum_sigclip_groups=minimum_sigclip_groups,
             only_use_ints=only_use_ints,
+            min_diffs_single_pass=min_diffs_single_pass,
         )
         # remove redundant bits in pixels that have jump flagged but were
         # already flagged as do_not_use or saturated.
@@ -362,6 +372,7 @@ def detect_jumps(
                     minimum_groups,
                     minimum_sigclip_groups,
                     only_use_ints,
+                    min_diffs_single_pass,
                 ),
             )
 
@@ -388,6 +399,7 @@ def detect_jumps(
                 minimum_groups,
                 minimum_sigclip_groups,
                 only_use_ints,
+                min_diffs_single_pass,
             ),
         )
         log.info("Creating %d processes for jump detection ", n_slices)
@@ -560,7 +572,7 @@ def flag_large_events(
     Nothing, gdq array is modified.
 
     """
-    log.info("Flagging large Snowballs")
+    log.info("Flagging Snowballs")
 
     n_showers_grp = []
     total_snowballs = 0
@@ -745,10 +757,10 @@ def find_last_grp(grp, ngrps, num_grps_masked):
     ----------
     grp : int
         The location of the shower
-    
+
     ngrps : int
         The number of groups in the integration
-    
+
     num_grps_masked : int
         The requested number of groups to be flagged after the shower
 
@@ -895,21 +907,24 @@ def find_faint_extended(
           emission.
       min_shower_area : int
           The minimum area for a group of pixels to be flagged as a shower.
-      inner: int
+      inner : int
           The inner radius of the ring_2D_kernal used for the convolution.
       outer : int
           The outer radius of the ring_2D_kernal used for the convolution.
       sat_flag : int
           The integer value of the saturation flag.
       jump_flag : int
           The integer value of the jump flag
-      ellipse_expand: float
+      ellipse_expand : float
           The relative increase in the size of the fitted ellipse to be
           applied to the shower.
-    num_grps_masked: int
-        The number of groups after the detected shower to be flagged as jump.
-    max_extended_radius: int
-        The upper limit for the extension of saturation and jump
+      num_grps_masked : int
+          The number of groups after the detected shower to be flagged as jump.
+      max_extended_radius : int
+          The upper limit for the extension of saturation and jump
+      minimum_sigclip_groups : int
+          The minimum number of groups to use sigma clipping.
+
 
     Returns
     -------