desi_proc_dashboard not showing all nights

[sbailey]

running desi_proc_dashboard on Himalayas starts out with

Searching /global/cfs/cdirs/desi/spectro/redux/himalayas for: ['20220613' '20220612' ... '20201212' '20201209']

however the final html dashboard only includes months 202105 through 202206, i.e. the SV nights aren't included.

Perhaps related: the original main survey processing was on Cori, while the SV nights were processed on Perlmutter, though in this case I'm running desi_proc_dashboard on Perlmutter and it is only showing the Cori nights. Or perhaps it is unrelated to Cori vs. Perlmutter and some other filter is going on.

[sbailey]

This might be a conflict with a cronjob not including SV nights stepping on the results of running it by hand. Check that first on cori21 crontab.

[sbailey]

I think I've seen a similar phenomenon with desi_zproc_dashboard. Check that too. i.e. get to a state where both the dashboard and zdashboard for himalayas includes all nights from all months covered in the production.

[akremin]

I turned off the cron job on cori21 and find that running desi_proc_dashboard on perlmutter does produce an html file with outputs for the months I request. I did not verify every individual night is present, but there are no missing months as mentioned in the issue.

For desi_proc_zdashboard I do see many missing months. My guess is that it has to do with the redshift jobs not being present in the processing tables. The SV months are shown, which were the ones run on perlmutter with redshifts as part of the standard pipeline. The main survey were run outside the main pipeline with desi_tile_redshifts, which don't add rows to the processing table. I will look into that.

[sbailey]

I believe all cases are understood. For desi_proc_dashboard it was a cronjob with a limited daterange overwriting the by-hand invocation outputs; for desi_proc_zdashboard it was due to redshift jobs submitted with desi_tile_redshifts which doesn't update the processing table (see #1952 for the workaround in desi_run_night so that we don't need to use desi_tile_redshifts for typical processing).

Although desi_zproc_dashboard could be updated to be based upon files on disk without looking at the processing table, that is a bit messier to know what is supposed to exist. I'm going to close this ticket and we can revive this in a separate focused ticket if needed.