Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates DRP for runs at Utah #41

Merged
merged 16 commits into from
Nov 28, 2023
Merged

Updates DRP for runs at Utah #41

merged 16 commits into from
Nov 28, 2023

Conversation

havok2063
Copy link
Collaborator

@havok2063 havok2063 commented Nov 10, 2023

This PR updates the DRP so it can more easily run at Utah. It reorganizes the quick_reduction function so it can be called directly, as quick_science_reduction. The original CLI quick-reduction is retained so users can continue to use it. It deprecates the old run_drp code and replaces it with a new run_drp that uses the quick_science_reduction function. It
also adds new functions for performing MJD checks against the daily data transfer, and writing status run files. Exposures without a tile_id get assigned a fake id of 1111, and output reductions are placed there.

It also includes the option of reducing the individual calibration files up through the detrending stage. This is currently turned off by default.

Manual test runs are at https://data.sdss5.org/sas/sdsswork/lvm/spectro/redux/master/ . I'm testing out a cron job at the moment to run a daily check every hour between 12-17:00 Utah time. It currently takes ~10 minutes to reduce a single exposure, using the default settings of quick_science_reduction. MJD 60256 had ~28 science exposures and took ~5 hours in total. This is simply iterating over each exposure in a single process. If we want to parallelize this, additional work needs to done to use the Utah CHPC cluster with the slurm package.

@havok2063 havok2063 added the enhancement New feature or request label Nov 10, 2023
@havok2063 havok2063 self-assigned this Nov 10, 2023
@havok2063 havok2063 marked this pull request as ready for review November 20, 2023 21:22
@havok2063
Copy link
Collaborator Author

@ajmejia this is ready to be merged.

Copy link
Contributor

@ajmejia ajmejia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had just a few minor comments. We can actually merge this branch as it is.

One more general comment is that I'm aiming at keeping the reduction of science frames and calibration frames as separated as possible. Right now the reduction for bias, dark, pixelflat, fiberflat and arcs is going through preprocessing and detrending, and that's fine. But after that what we do to the frames may differ a lot depending on the type of frame and calibration sequence that we have.

from lvmdrp.functions.imageMethod import preproc_raw_frame
from lvmdrp.functions.skyMethod import configureSkyModel_drp
from lvmdrp.utils.metadata import get_frames_metadata, get_master_metadata

from lvmdrp.functions.run_quickdrp import quick_reduction
from lvmdrp.functions.run_quickdrp import quick_science_reduction
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be renamed to science_reduction throughout, just to avoid confusions. It's not running a quick pipeline anymore.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can certainly rename this, but if it's no longer the case of being a quick pipeline, then we should reorganize the modules.

But this isn't doing optimal extraction, right? Nor Amy's full sky subtraction. Or a complete flux calibration? Presumably we will retain both quick and full modes once those are in place?

@@ -222,6 +219,8 @@ def quick_reduction(expnum: int, use_fiducial_master: bool, skip_sky_subtraction
mlsf_path = path.full("lvm_master", drpver=drpver, kind=f"mlsf_{lamps}", **masters["lsf"].to_dict())
mflat_path = path.full("lvm_master", drpver=drpver, kind="mfiberflat", **masters["fiberflat"].to_dict())

log.info(f'--- Starting science reduction of raw frame: {rsci_path}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this line to inside of the preprocessing routine. In general I'm trying to keep stuff like "doing X to file path/to/file" and "writing Xed output to path/to/file" inside the routine that does X. It keeps things a bit more tight and self-contained.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to have a general log line that indicates the start of a single file reduction, not any particular step.

path = root / f'lvm-drp-{tileid}-{mjd}.{status}'
return path.exists()


def update_error_file(tileid: int, mjd: int, expnum: int, error: str,
reset: bool = False):
"""_summary_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function is easy to read, but you may want to fill in this placeholder docstring.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

@ajmejia ajmejia merged commit 2fbb1fe into master Nov 28, 2023
@ndrory
Copy link
Contributor

ndrory commented Nov 28, 2023 via email

@havok2063 havok2063 deleted the utah branch December 12, 2023 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants