Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the full loop for MTL #684

Merged
merged 13 commits into from Mar 18, 2021
Merged

Implement the full loop for MTL #684

merged 13 commits into from Mar 18, 2021

Conversation

geordie666
Copy link
Contributor

This PR implements the full loop of constructing a zcat by reading "done" tiles from a specified directory, determining which "done" tiles are yet to be processed, updating ledgers, and creating and updating an "MTL tile file" to record which tiles have been processed through MTL. New features include:

  • Modify the ledgers based on any new (i.e. previously unprocessed) tiles in a zcat directory.
  • Create and update an "MTL tile file" which tracks which tiles have been processed by MTL.
  • Read the "standard" tile file to determine the observing conditions for each tile, and only process a given tile for the appropriate observing conditions.
    • This should also only process tiles if they are in the appropriate tile file for a given survey flavor (sv2, main, etc.).
  • An option to use the ledgers themselves to update NUMOBS instead of expecting NUMOBS to be in the zcat.
    • Currently this option is turned on, but it will be easy to turn off if we decide to track NUMOBS (e.g. via NUMTILE) in the zcat.
  • A command-line script to execute the full loop. Each of the required input directories can be passed or can be environment variables.
    • An example is: run_mtl_loop DARK --survey sv2 --zcatdir $ZCAT_DIR --mtldir $MTL_DIR --tilefn $TILE_FN

A number of refinements are likely to be needed that build on this PR, as:

  • We haven't yet converged on the data model and full directory path for the zbest files.
  • Eleanor will eventually want to drop in a different make_zcat() function to replace my simple make_zcat_rr_backstop().
  • We haven't yet determined whether we will use NUMTILE from the spectroscopic pipeline outputs, or if we will let the ledger update NUMOBS itself.
  • We will need to update priorities and NUMOBS more carefully based on ZWARN

But this is likely enough infrastructure to have a first crack at the 0.1% survey.

@coveralls
Copy link

coveralls commented Mar 16, 2021

Coverage Status

Coverage decreased (-0.5%) to 59.648% when pulling 71fd9a5 on ADM-MTL-loop into 4fc5e72 on master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants