Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finalize the MTL logic for reprocessed tiles #780

Merged
merged 24 commits into from Dec 14, 2021
Merged

Conversation

geordie666
Copy link
Contributor

This PR extends #774 by including the actual, final logic for reprocessing tiles. This logic is to:

  • Assemble all previous observations that touch the set of reprocessed tiles including any "new" observations from the reprocessed tiles.
  • Find the most recent observations for unique combinations of TILEID + TARGETID and store these in a redshift catalog (which I'll refer to as the "all-zcat" below).
  • Determine the order in which tiles were originally processed and loop through the tiles in this original order:
    • Start with the UNOBS state for a given TARGETID.
    • Update the UNOBS state (progressively, in tile order) using entries from the "all-zcat" for a given TARGETID.
    • Hence, recover the final state for each TARGETID with the reprocessed observations substituted for the "orginal" observations.
  • The full progression of states for each TARGETID is then appended to the ledgers, including BAD observations (which, previously, never appeared in the ledgers).

This PR also includes:

  • During reprocessing, TIMESTAMPs are mocked up in advance for speed.
    • i.e. instead of having to delay the code by 1 second for each tile so as to ensure a unique TIMESTAMP.
  • The numobsfromzcat flag is deprecated as a user-specified option when executing run_mtl_loop.
    • as we have always, in actuality, retrieved NUMOBS from the ledgers themselves rather than the redshift catalog.
    • it seemed increasingly fragile to retain an option we've never used.

… tiles. This could be used to ease the strain on committing MTL updates to svn.
… the ledgers share the TIMESTAMP in the mtl-done-tiles files
…n a previous tile we can't batch. Any delay will be handled in archiving not MTL
…elay, as archiving can add any needed delay instead
…om the ledger. We always use that mode, so it's fragile to make it optional
@geordie666
Copy link
Contributor Author

A full set of example reprocessed ledgers is available in /global/cscratch1/sd/adamyers/fakesurveyops4.

@geordie666 geordie666 merged commit 303be12 into master Dec 14, 2021
@geordie666 geordie666 deleted the ADM-reproc-logic branch December 14, 2021 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant