Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nightly bias and bad column flagging into desi_run_night #1465

Merged
merged 5 commits into from Oct 29, 2021

Conversation

akremin
Copy link
Member

@akremin akremin commented Oct 29, 2021

Overview

This is the final PR in the effort to add the bias modeling and bad column identification into the pipeline. This pull request incorporates darks into the reprocessing code, desi_run_night.

Most of the underlying functions were updated yesterday. This primarily edits the coordination script desi_run_night so that it keeps and submits dark jobs. While doing so I also implemented a small feature that moves the calibrations to the start of the exposures so that cals are always processed first, in order of: 1) the first 300s dark, 2) arcs+flats, 3) science exposures.

I also fixed a bug meant to catch darks of the wrong EXPTIME.

Reprocessing Test

I've tested this on 20211020 with multiple darks and it runs the first dark and skips the subsequent darks as expected. I tested it on one "typical" night, 20211027, and it did the proper thing there as well. Finally I ran a test night of jobs, stopping halfway through the sciences. The dependencies in the queue were correctly set and the jobs ran properly. I verified that the biases were used and the badcolumn files were also used. This was night 20211026 in the test prod.

commands:
desi_run_night --dry-run-level=1 -n 20211020 &>test_reprocdarks_20211020.log &
desi_run_night -n 20211026
desi_run_night --dry-run-level=1 -n 20211027

The prod used for testing:
/global/cfs/cdirs/desi/users/kremin/spectro/redux/reprocdarks

daily processing (re-) Test

In addition, since I changed two thing in the daily processing script, I reran in dry-run mode:
desi_daily_proc_manager --override-night=20211020 --dry-run-level=1 --ignore-cori-node

and it once again did the right thing. The prod used for that test was:
/global/cfs/cdirs/desi/users/kremin/spectro/redux/usedarks

Note: I had already tested that night in the PR yesterday. Those results are in *.orig.csv versions of the output tables, while the standard names are the new ones produced for this test. I chose that night because it had two darks, which tests slightly more of the new coding logic.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.01%) to 26.316% when pulling 0ef4d2d on nightlybias-reproc into 0327328 on master.

@sbailey
Copy link
Contributor

sbailey commented Oct 29, 2021

Looks good, thanks. Merge when ready. I will trust your tests and the upcoming run to catch any remaining issues if any.

@akremin akremin merged commit 8d2f7f0 into master Oct 29, 2021
@akremin akremin deleted the nightlybias-reproc branch October 29, 2021 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants