submit cumulative redshifts only for lastnight of each tile

[sbailey]

This PR adds support to desi_run_night to submit cumulative redshifts only for tiles for which the current night is the last night that they were observed. This allows them to be tracked in the processing tables (see issue #1952) while not submitting cumulative redshift jobs for nights earlier than the LASTNIGHT for each tile.

I did not implement an option to override this and submit cumulative redshifts for every tile, even if they have future observations (current behavior). The code is already complex from supporting tilenight, non-tilenight, productions, and daily-like "is this the last exposure for this tile or the end of the night?" style logic, and I didn't want to add additional options that we may never use and then feel obligated to support those in any future refactors. If we find we need that functionality, we can add it later.

Tests

I tested on the following nights, to cover several cases:

• 20211216 - ends with a science exposure
• 20220602 - ends with a full set of arcs+flats+zeros+darks
• 20220603 - ends with zeros+darks but the science exposures are the last that would be processed

export DESI_SPECTRO_REDUX=/global/cfs/cdirs/desi/users/sjbailey/spectro/redux
export SPECPROD=zproc
cd $DESI_SPECTRO_REDUX/$SPECPROD
svn checkout https://desi.lbl.gov/svn/data/exposure_tables

desi_run_night --dry-run-level=2 -n 20211216 --z-submit-types pernight,cumulative &> 20211216-submit.log
desi_job_graph -n 20211216
...

The resulting jobgraphs in https://data.desi.lbl.gov/desi/users/sjbailey/spectro/redux/zproc/run/jobgraph/ show that all tiles get perexp redshift jobs, but only some tiles get cumulative redshift jobs. I used code like the following to identify which tiles are supposed to get cumulative jobs:

from desispec.workflow.redshifts import read_minimal_exptables_columns
exps = read_minimal_exptables_columns()

night = 20220603
tiles = np.unique(exps["TILEID"][exps["NIGHT"] == night])
for t in tiles:
    jj = exps["TILEID"] == t
    lastnight = np.max(exps["NIGHT"][jj])
    if lastnight == night:
        print(f"tile {t} - ok to submit")
    else:
        print(f"tile {t} {lastnight=}")

I repeated with desi_run_night --no-use-tilenight ... to make sure that variant of the logic works too.

[akremin]

The code looks great and the tests are thorough. This will be susceptible to the year-boundary issue outlined in #1943, but will be automatically accounted for when that is resolved, so nothing needs to be done with that here. I will do more testing once this and the PR resolving #1943 are both implemented to ensure we don't have any corner cases that have snuck through. For now I will approve and merge this.

[akremin]

The failed tests are related to a scipy issue, not an issue with this PR. The tests are now fixed in the daily branch thanks to desihub/desimodel#164 and desispec #1957 , and fixed in main thanks to #1958.