Remove unnecessary output files from bias/readnoise monitors

[bsunnquist]

This PR addresses #930, to remove fits files from the bias/readnoise monitors that are no longer needed once processing completes, to save memory space on our servers. It also removes the uncal files if pipeline processing fails so they aren't just hanging around taking up space.

The web app only needs the output pngs and database entries to construct the webpages, which is why it's fine to remove these.

[pep8speaks]

Hello @bsunnquist, Thank you for updating !

• In the file jwql/instrument_monitors/common_monitors/bias_monitor.py:

Line 45:1: E402 module level import not at top of file
Line 46:1: E402 module level import not at top of file
Line 47:1: E402 module level import not at top of file
Line 48:1: E402 module level import not at top of file
Line 49:1: E402 module level import not at top of file
Line 51:1: E402 module level import not at top of file
Line 52:1: E402 module level import not at top of file
Line 53:1: E402 module level import not at top of file
Line 54:1: E402 module level import not at top of file
Line 55:1: E402 module level import not at top of file
Line 56:1: E402 module level import not at top of file
Line 57:1: E402 module level import not at top of file
Line 58:1: E402 module level import not at top of file
Line 59:1: E402 module level import not at top of file
Line 63:1: E303 too many blank lines (3)
Line 347:13: E128 continuation line under-indented for visual indent
Line 347:42: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 383:13: E722 do not use bare 'except'

• In the file jwql/instrument_monitors/common_monitors/readnoise_monitor.py:

Line 47:1: E402 module level import not at top of file
Line 48:1: E402 module level import not at top of file
Line 49:1: E402 module level import not at top of file
Line 50:1: E402 module level import not at top of file
Line 52:1: E402 module level import not at top of file
Line 53:1: E402 module level import not at top of file
Line 54:1: E402 module level import not at top of file
Line 55:1: E402 module level import not at top of file
Line 56:1: E402 module level import not at top of file
Line 57:1: E402 module level import not at top of file
Line 58:1: E402 module level import not at top of file
Line 59:1: E402 module level import not at top of file
Line 60:1: E402 module level import not at top of file
Line 61:1: E402 module level import not at top of file
Line 62:1: E402 module level import not at top of file
Line 63:1: E402 module level import not at top of file
Line 64:1: E402 module level import not at top of file
Line 357:61: E226 missing whitespace around arithmetic operator
Line 357:110: E226 missing whitespace around arithmetic operator
Line 360:61: E226 missing whitespace around arithmetic operator
Line 360:110: E226 missing whitespace around arithmetic operator
Line 372:33: E226 missing whitespace around arithmetic operator
Line 389:13: E128 continuation line under-indented for visual indent
Line 389:42: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 424:13: E722 do not use bare 'except'
Line 441:13: E265 block comment should start with '# '
Line 442:13: E265 block comment should start with '# '
Line 461:13: E722 do not use bare 'except'
Line 609:67: E226 missing whitespace around arithmetic operator
Line 609:70: E226 missing whitespace around arithmetic operator
Line 611:66: E226 missing whitespace around arithmetic operator

Comment last updated at 2022-04-22 20:48:37 UTC

[mfixstsci]

@bsunnquist thanks for submitting this! I was curious about the removing the uncal files if the pipeline fails. Does that mean future monitor runs will pick up these files and attempt to process them again?

[bsunnquist]

@mfixstsci Yeah that's correct, as long as the filename isn't in the monitor database (i.e. completed processing) it will keep trying each day. I figured it was best to delete the uncal's between runs though so they aren't hanging around, because like we saw recently, if the pipeline fails this can result in a lot of large data files taking up space on our servers

[bhilbert4]

Are there any other pipeline products that may need to be removed? I'm thinking about e.g. trapsfilled, if the pipeline is run far enough.

[bsunnquist]

I don't think so - these just run dq/superbias/refpix

[bhilbert4]

This PR looks good to me. @mfixstsci do you agree?

[mfixstsci]

@bhilbert4 it does look good, I forgot to fully submit my review 😓 I wanted to ask @bsunnquist a question about the readnoise file we are no longer generating at the end. Do we want to make some sort of utils.py function to allow users to make the file using our API?