Better functionality for fiber-based ToOs and ledger overrides

[geordie666]

This PR updates how ToOs with TOO_TYPE=="FIBER" are handled. It also adds some new functionality for the override ledgers that we use to modify targets that are already in the MTL ledgers.

Updates to the ToO functionality include:

• New ToOs are now appended to the ledger rather than fully overwriting it.
  • This preserves TIMESTAMPs for earlier ToOs.
• A separate ToO-fiber.ecsv file is now created for TOO_TYPE==FIBER ToOs.
  • We'll retain the ToO.ecsv file for backwards-compatibility.
  • The ToO.ecsv file will include all of the ToO observations to-date in the Main Survey, which have solely been TOO_TYPE==TILE observations.
  • It's likely, in the future, that any TOO_TYPE=TILE observations will be scheduled through the more-flexible tertiary process.
• TOO_PRIO==HI observations are now forbidden for TOO_TYPE==FIBER ToOs in the Main Survey.
  • This addresses Never let high priority TOOs for fiber overrides #794.

Updates to the override-ledger functionality include:

• New tracking files called scnd-mtl-done-overrides.ecsv and mtl-done-overrides.ecsv are now created/updated when target states are forced into the MTL ledgers using the override process.
  • These resemble the more familiar scnd-mtl-done-tiles.ecsv and mtl-done-tiles.ecsv files, but have zeroed-out tile-based information (as when we force override information into the MTL ledgers, we don't use any tile-based redshifts).
  • These done-overrides files can be used by the alt MTLs to determine when forced overrides occurred, if needed.
• When forcing MTL overrides, we always process all of the ledgers, not just new overrides.
  • This is quick and harmless, but makes it easier for the alt MTLs to reproduce the process.

[geordie666]

@araichoor is checking the updates to the ToO mechanism from the fiberassign perspective. I'll likely merge this and cut a new desitarget tag once Anand is satisfied that everything gels.

[coveralls]

Coverage decreased (-0.07%) to 56.396% when pulling 1d7d663 on ADM-ToO-over into 1c7edc0 on master.

[araichoor]

I ve coded up the twin PR in fiberassign, which works fine for the main survey tiles.

one question though, when designing ToO-dedicated tile (maybe deprecated with the tertiary programs):

in the current fiberassign code, for a ToO-dedicated tile, will read ToO.ecsv and select both TOO_TYPE="TILE" and "FIBER" targets:
https://github.com/desihub/fiberassign/blob/854abc4d8c09250bb88855ee55be5d89a25fef38/py/fiberassign/fba_launch_io.py#L1637-L1638
with the current scheme, for that case, the code will read the ToO.ecsv file, and hence just grab the TOO_TYPE="TILE" targets.
(Adam, actually that the initial reason of my question on slack if ToO.ecsv would contain both TOO_TYPE="TILE" and "FIBER" targets; but I then forgot...)

maybe it s fine, but I just wanted to mention.

[geordie666]

I think the case you describe is probably fine, Anand. I suspect we'll want to use the tertiary process in the future instead of TOO_TYPE="TILE". If not, then the ToO folks would just have to supply their targets as "TILE" targets instead of "FIBER" targets.

I'm going to merge this soon and make a new tag of desitarget. I have lots of meetings today, but I'll try to have a tag ready as soon as possible.

[araichoor]

ok perfect then, thanks!