New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Finalize the MTL logic for reprocessed tiles #780
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… that reprocessed tiles can appear twice.
… tiles. This could be used to ease the strain on committing MTL updates to svn.
… the ledgers share the TIMESTAMP in the mtl-done-tiles files
…of the value on the specific tile
…n a previous tile we can't batch. Any delay will be handled in archiving not MTL
… to be processed.
…elay, as archiving can add any needed delay instead
…because reprocessed tiles COULD overlap
…or each TARGETID/TILEID for reprocessing
…ich the tiles were processed
…need to add delays
…om the ledger. We always use that mode, so it's fragile to make it optional
A full set of example reprocessed ledgers is available in |
…tate, so we need to specify when we mean the ORIGINAL UNOBS state
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends #774 by including the actual, final logic for reprocessing tiles. This logic is to:
TILEID
+TARGETID
and store these in a redshift catalog (which I'll refer to as the "all-zcat" below).UNOBS
state for a givenTARGETID
.UNOBS
state (progressively, in tile order) using entries from the "all-zcat" for a givenTARGETID
.TARGETID
with the reprocessed observations substituted for the "orginal" observations.TARGETID
is then appended to the ledgers, includingBAD
observations (which, previously, never appeared in the ledgers).This PR also includes:
TIMESTAMP
s are mocked up in advance for speed.TIMESTAMP
.numobsfromzcat
flag is deprecated as a user-specified option when executingrun_mtl_loop
.NUMOBS
from the ledgers themselves rather than the redshift catalog.