Improvements to short flat handling in daily pipeline #1517
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a corner case when re-loading flats after a script crash where upon restarting the script the 1s short flat will be included with the other flats for use in producing the
nightlyflat
's. This corrects that by only adding a flat if it is over 100s, but still allowing the loop to continue looking for additional flats of the correct type.This also improves the saving of tables by writing out after joint fitting. Previously the tables were only written out after a single exposure processing, so some jobs could be submitted but not saved to the table on disk before a crash. Now we write out after each joint fit.
I tested this using
dry-run-level=2
, which produces the tables but doesn't submit jobs. I killed the script after joint fits and found the joint jobs were written out before the script crashed (the desired new behavior). I also confirmed that the 1s flat is no longer included in the list of flats used fornightlyflat
's if the script re-initializes from the tables after a crash (i.e. the corner case is now resolved).