Make Readnoise Monitor less memory-intensive #922

bsunnquist · 2022-04-08T15:13:05Z

This PR does the following

switch to perform the readnoise calculation in smaller slices to reduce memory pressure
makes the plot limits work better with the ranges in values we're seeing (applied same change to bias monitor too, while I was at it)
less stringent minimum groups required to calculate readnoise, to maximize the number of data points
increased the lookback time, so these changes would be applied to all of the new darks we received

The first point is the main one - it addresses the memory issues noted in #905 , and (after the jwst pipeline is switched to jwst1.4.3) should also solve #900.

…igent

pep8speaks · 2022-04-08T15:13:08Z

Hello @bsunnquist, Thank you for updating !

In the file jwql/instrument_monitors/common_monitors/readnoise_monitor.py:

Line 47:1: E402 module level import not at top of file
Line 48:1: E402 module level import not at top of file
Line 49:1: E402 module level import not at top of file
Line 50:1: E402 module level import not at top of file
Line 52:1: E402 module level import not at top of file
Line 53:1: E402 module level import not at top of file
Line 54:1: E402 module level import not at top of file
Line 55:1: E402 module level import not at top of file
Line 56:1: E402 module level import not at top of file
Line 57:1: E402 module level import not at top of file
Line 58:1: E402 module level import not at top of file
Line 59:1: E402 module level import not at top of file
Line 60:1: E402 module level import not at top of file
Line 61:1: E402 module level import not at top of file
Line 62:1: E402 module level import not at top of file
Line 63:1: E402 module level import not at top of file
Line 64:1: E402 module level import not at top of file
Line 357:61: E226 missing whitespace around arithmetic operator
Line 357:110: E226 missing whitespace around arithmetic operator
Line 360:61: E226 missing whitespace around arithmetic operator
Line 360:110: E226 missing whitespace around arithmetic operator
Line 372:33: E226 missing whitespace around arithmetic operator
Line 389:13: E128 continuation line under-indented for visual indent
Line 389:42: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 424:13: E722 do not use bare 'except'
Line 460:13: E722 do not use bare 'except'
Line 608:67: E226 missing whitespace around arithmetic operator
Line 608:70: E226 missing whitespace around arithmetic operator
Line 610:66: E226 missing whitespace around arithmetic operator

Comment last updated at 2022-04-13 16:46:16 UTC

bsunnquist · 2022-04-08T16:00:33Z

@bhilbert4 this is ready for review whenever. It should solve all of the ongoing readnoise monitor issues (assuming also the jwst pipeline is updated to 1.4.3)

jwql/instrument_monitors/common_monitors/readnoise_monitor.py

mfixstsci · 2022-04-11T17:46:44Z

@bsunnquist thank you for the submission, I went ahead and had a few questions/comments.

bsunnquist · 2022-04-12T18:55:10Z

thanks @mfixstsci , did your review post? I don't see any questions, just that one comment so far

mfixstsci · 2022-04-11T15:33:40Z

jwql/instrument_monitors/common_monitors/readnoise_monitor.py

@@ -542,16 +548,23 @@ def run(self):
            siaf = Siaf(self.instrument)
            possible_apertures = list(siaf.apertures)

+            # Get the minimum groups needed to calculate readnoise for this instrument. MIRI
+            # removes the first five and last group before calculating the readnoise, so needs extra.
+            if instrument == 'miri':


Were all minimum groups set to a higher value previously?

yes, and it missed the cases where there were lots of ints but low groups, which should still be able to give a good cds readnoise measure. Though this comment reminded me of something, so I made another tweak to this logic in a new push.

mfixstsci · 2022-04-11T15:36:02Z

jwql/website/apps/jwql/monitor_pages/monitor_bias_bokeh.py

@@ -181,5 +181,11 @@ def update_mean_bias_figures(self):
                if len(bias_vals) != 0:
                    self.refs['mean_bias_xr_amp{}_{}'.format(amp, kind)].start = expstarts.min() - timedelta(days=3)
                    self.refs['mean_bias_xr_amp{}_{}'.format(amp, kind)].end = expstarts.max() + timedelta(days=3)
-                    self.refs['mean_bias_yr_amp{}_{}'.format(amp, kind)].start = min(x for x in bias_vals if x is not None) - 20
-                    self.refs['mean_bias_yr_amp{}_{}'.format(amp, kind)].end = max(x for x in bias_vals if x is not None) + 20
+                    min_val, max_val = min(x for x in bias_vals if x is not None), max(x for x in bias_vals if x is not None)


Thanks for tackling the same issue with the bias monitor as well.

mfixstsci · 2022-04-13T13:08:30Z

@bsunnquist I guess I didn't hit the big green button 😓 let me know if you see it now.

bsunnquist · 2022-04-13T16:34:26Z

thanks @mfixstsci I just made one more minor tweak to the minimum group/int logic, but let me know if you see any issues. This latest strategy will make sure images with lots of ints/low groups will still get processed, but also ensure the total number of these cds frames is still large enough to give meaningful measurements

bsunnquist · 2022-04-13T16:46:59Z

and one more tweak, to push back the lookback time a little more to make sure it picks up the earliest dark data

mfixstsci

Looks good to me!

calculate readnoise in smaller slices; making plot limits more intell…

53a175d

…igent

bsunnquist added bug enhancement Monitor High Priority labels Apr 8, 2022

bsunnquist self-assigned this Apr 8, 2022

pep8 whitespace

735deb0

mfixstsci self-requested a review April 11, 2022 15:25

mfixstsci reviewed Apr 11, 2022

View reviewed changes

jwql/instrument_monitors/common_monitors/readnoise_monitor.py Show resolved Hide resolved

mfixstsci self-requested a review April 11, 2022 20:58

mfixstsci approved these changes Apr 13, 2022

View reviewed changes

changing minimum cds frame strategy

0f405b8

pushing back lookback to get earliest commish darks

def8979

mfixstsci approved these changes Apr 13, 2022

View reviewed changes

mfixstsci merged commit 2980a5f into spacetelescope:develop Apr 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Readnoise Monitor less memory-intensive #922

Make Readnoise Monitor less memory-intensive #922

bsunnquist commented Apr 8, 2022

pep8speaks commented Apr 8, 2022 •

edited

bsunnquist commented Apr 8, 2022

mfixstsci commented Apr 11, 2022

bsunnquist commented Apr 12, 2022

mfixstsci Apr 11, 2022

bsunnquist Apr 13, 2022

mfixstsci Apr 11, 2022

mfixstsci commented Apr 13, 2022

bsunnquist commented Apr 13, 2022 •

edited

bsunnquist commented Apr 13, 2022

mfixstsci left a comment

Make Readnoise Monitor less memory-intensive #922

Make Readnoise Monitor less memory-intensive #922

Conversation

bsunnquist commented Apr 8, 2022

pep8speaks commented Apr 8, 2022 • edited

Comment last updated at 2022-04-13 16:46:16 UTC

bsunnquist commented Apr 8, 2022

mfixstsci commented Apr 11, 2022

bsunnquist commented Apr 12, 2022

mfixstsci Apr 11, 2022

Choose a reason for hiding this comment

bsunnquist Apr 13, 2022

Choose a reason for hiding this comment

mfixstsci Apr 11, 2022

Choose a reason for hiding this comment

mfixstsci commented Apr 13, 2022

bsunnquist commented Apr 13, 2022 • edited

bsunnquist commented Apr 13, 2022

mfixstsci left a comment

Choose a reason for hiding this comment

pep8speaks commented Apr 8, 2022 •

edited

bsunnquist commented Apr 13, 2022 •

edited