header keyword not found: KeyError: "Keyword 'STD1EXP' not found."

[havok2063]

Trying to reduce exposure 3507 from 60178 crashes during flux calibration. This exposure is missing all the STD keywords, likely as they weren't added in yet. The pipeline should be able reduce data from all MJDs successfully. At least from 60177 onwards, and ideally from 60142 onwards. Or at least fail cleanly with a message why it can't proceed.

  File "/Users/Brian/Work/github_projects/sdss/lvm/lvmdrp/python/lvmdrp/functions/run_quickdrp.py", line 280, in quick_reduction
    flux_tasks.apply_fluxcal(in_rss=sci_path, out_rss=sci_path)
  File "/Users/Brian/Work/github_projects/sdss/lvm/lvmdrp/python/lvmdrp/functions/fluxCalMethod.py", line 104, in apply_fluxcal
    exptime = rss._header[f"{std_hd[:-3]}EXP"]
  File "/Users/Brian/anaconda3/envs/lvmdrp/lib/python3.9/site-packages/astropy/io/fits/header.py", line 169, in __getitem__
    card = self._cards[self._cardindex(key)]
  File "/Users/Brian/anaconda3/envs/lvmdrp/lib/python3.9/site-packages/astropy/io/fits/header.py", line 1819, in _cardindex
    raise KeyError(f"Keyword {keyword!r} not found.")
KeyError: "Keyword 'STD1EXP' not found."

[havok2063]

Or the other alternative is to ignore all MJDs before the required header keys for flux calibration were in place. In which case, can you give me the minimum MJD where that is the case?

[ndrory]

Brian, this looks like a bug, I'm sure we can fix it once Alfredo is back from vacation and moved to Chile. N. On Nov 7, 2023, at 10:05 AM, Brian Cherinka ***@***.***> wrote: Or the other alternative is to ignore all MJDs before the required headers for flux calibration was in place. In which case, can you give me the minimum MJD where that is the case? — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJXJJ7ISWX74OZ6JNYWBXDLYDJL5HAVCNFSM6AAAAAA7BM5PBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJZGAZTOMJRGA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

…

-- Niv Drory — McDonald Observatory / Dept. of Astronomy The University of Texas at Austin Tel: +1 512 471 6197 http://www.as.utexas.edu/~drory

[EvelynJ]

Hi Brian I would prefer to avoid this alternative option if possible… A lot of the early science targets were observed from around MJD 60178 (before that we still had issues with the k-mirror tracking), so if we only reduce data from the MJD that the headers have the correct information we lose a lot of that data. I wonder if there’s a way to reproduce the standard stars selected for each science pointing from the tile databases? If I remember correctly, the database already has details of the assigned sky and std pointings for each survey tile, and when we observed the early science targets the software identifies the nearest survey tile and uses those sky and std pointings. So maybe we could use that information to update the headers for the DRP? I guess Jose or John would be the best people to confirm if my memory is correct on how the std and sky targets are selected for the science targets, but I’m not sure how easy it would then be to update all the headers to run the DRP. Cheers Evelyn

On 7 Nov 2023, at 13:05, Brian Cherinka ***@***.***> wrote: Or the other alternative is to ignore all MJDs before the required headers for flux calibration was in place. In which case, can you give me the minimum MJD where that is the case? — Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWIBTIBIUDT7GSURFZ6QATYDJL5HAVCNFSM6AAAAAA7BM5PBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJZGAZTOMJRGA>. You are receiving this because you are subscribed to this thread.

=================================== Dr. Evelyn Johnston Instituto de Estudios Astrofísicos Facultad de Ingeniería y Ciencias Universidad Diego Portales (UDP) Av. Ejército 441 Santiago, Chile https://evelynjohnston.space/ ***@***.*** ***@***.*** +56 2 2213 0356

[havok2063]

That all makes sense to me. We can wait until Alfredo is back to fix it. We do have a mechanism in place to fix headers with the hdrfix files in lvmcore, that the pipeline already understands. The pipeline uses these files to correct the raw headers during the metadata collection before reductions. We could put that info there. Although with 15 standards and 8 keywords per standard, it might be a bit clunky to add for all those mjds.

[ndrory]

Brian, we have to skip the calibration for the cases where the header info is incomplete since we do not have the exposure times for the standards N. On Nov 7, 2023, at 10:45 AM, Brian Cherinka ***@***.***> wrote: That all makes sense to me. We can wait until Alfredo is back to fix it. We do have a mechanism in place to fix headers with the hdrfix<https://github.com/sdss/lvmcore/tree/master/hdrfix> files in lvmcore, that the pipeline already understands. The pipeline uses these files to correct the raw headers during the metadata collection before reductions. We could put that info there. Although with 15 standards and 8 keywords per standard, it might be a bit clunky to add for all those mjds. — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJXJJ7NJYCVWR4HQZXYOY3LYDJQUNAVCNFSM6AAAAAA7BM5PBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJZGE3TSMZTGU>. You are receiving this because you commented.Message ID: ***@***.***>

…

-- Niv Drory — McDonald Observatory / Dept. of Astronomy The University of Texas at Austin Tel: +1 512 471 6197 http://www.as.utexas.edu/~drory

[havok2063]

That's fine too. In that case, I think we should try to avoid having lvmCFrame files that are flux-calibrated for some MJDs but not for others. I think we want to try to be consistent with what the data products mean/contain. We may want to create a separate output file for before/after flux calibration. For those MJDs where we can't flux calibrate, the pipeline ends there, updating some header flag about what stage of the pipeline it stopped at and why.

[ajmejia]

Fixed this in commit:

b5fb588

The only caveat is that DRP runs of exposures without std fibers metadata will produce lvmCFrames in electrons. I'll address this in a separated PR.