Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

track cumulative redshift jobs in processing tables #1952

Open
sbailey opened this issue Jan 4, 2023 · 1 comment
Open

track cumulative redshift jobs in processing tables #1952

sbailey opened this issue Jan 4, 2023 · 1 comment
Labels

Comments

@sbailey
Copy link
Contributor

sbailey commented Jan 4, 2023

For Himalayas, we submitted cumulative redshift jobs post-facto using

desi_tile_redshifts -t TILEID -g cumulative --run_zmtl

after first processing all exposures through cframes. This is tedious and error prone, especially since these jobs are not tracked via the processing tables to make them visible to desi_proc_dashboard, desi_job_graph, and desi_resubmit_queue_failures.

Two options to improve get these into the processing tables:

  1. desi_run_night already allows submitting cumulative redshifts and adding them to the processing table, but it doesn't know about whether this is the last night for a tile and whether a cumulative redshift job should be submitted. Add hooks to look at future exposure tables to decide which tiles should get cumulative redshifts for this night.
  2. add hooks to desi_tile_redshifts to also update the processing tables.

After discussions with @akremin, we think we want both, but should start with (1) since that would allow us to submit cumulative redshift jobs at the same time as everything else. The new desi_zproc will handle cross-night dependencies and exit with an error if a required previous night isn't finished yet. This isn't quite as clean as full cross-night job dependencies, but it gives the flexibility of submitting nights in any order and then cleaning up failures with desi_resubmit_queue_failures.

This isn't strictly needed for Iron, but it would make that production go much more smoothly so I'm going to add it to the Iron project. The alternative is "be more careful in the first place and resubmit jobs by hand as needed", which will likely be more work than implementing this feature in the first place.

@sbailey
Copy link
Contributor Author

sbailey commented Jan 6, 2023

Part 1 is implemented (desi_run_night updates) by PR #1955. Part 2 is still do to, but isn't critical for Iron so I will flag it as done for that project while leaving the ticket open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

No branches or pull requests

1 participant