You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
after first processing all exposures through cframes. This is tedious and error prone, especially since these jobs are not tracked via the processing tables to make them visible to desi_proc_dashboard, desi_job_graph, and desi_resubmit_queue_failures.
Two options to improve get these into the processing tables:
desi_run_night already allows submitting cumulative redshifts and adding them to the processing table, but it doesn't know about whether this is the last night for a tile and whether a cumulative redshift job should be submitted. Add hooks to look at future exposure tables to decide which tiles should get cumulative redshifts for this night.
add hooks to desi_tile_redshifts to also update the processing tables.
After discussions with @akremin, we think we want both, but should start with (1) since that would allow us to submit cumulative redshift jobs at the same time as everything else. The new desi_zproc will handle cross-night dependencies and exit with an error if a required previous night isn't finished yet. This isn't quite as clean as full cross-night job dependencies, but it gives the flexibility of submitting nights in any order and then cleaning up failures with desi_resubmit_queue_failures.
This isn't strictly needed for Iron, but it would make that production go much more smoothly so I'm going to add it to the Iron project. The alternative is "be more careful in the first place and resubmit jobs by hand as needed", which will likely be more work than implementing this feature in the first place.
The text was updated successfully, but these errors were encountered:
Part 1 is implemented (desi_run_night updates) by PR #1955. Part 2 is still do to, but isn't critical for Iron so I will flag it as done for that project while leaving the ticket open.
For Himalayas, we submitted cumulative redshift jobs post-facto using
after first processing all exposures through cframes. This is tedious and error prone, especially since these jobs are not tracked via the processing tables to make them visible to desi_proc_dashboard, desi_job_graph, and desi_resubmit_queue_failures.
Two options to improve get these into the processing tables:
After discussions with @akremin, we think we want both, but should start with (1) since that would allow us to submit cumulative redshift jobs at the same time as everything else. The new desi_zproc will handle cross-night dependencies and exit with an error if a required previous night isn't finished yet. This isn't quite as clean as full cross-night job dependencies, but it gives the flexibility of submitting nights in any order and then cleaning up failures with desi_resubmit_queue_failures.
This isn't strictly needed for Iron, but it would make that production go much more smoothly so I'm going to add it to the Iron project. The alternative is "be more careful in the first place and resubmit jobs by hand as needed", which will likely be more work than implementing this feature in the first place.
The text was updated successfully, but these errors were encountered: