Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-18739: allow ingesting defects with a slim wrapper on ingestCalibs #295

Merged
merged 12 commits into from
Jun 21, 2019
Merged

DM-18739: allow ingesting defects with a slim wrapper on ingestCalibs #295

merged 12 commits into from
Jun 21, 2019

Commits on Jun 21, 2019

  1. Remove mention of defects in ingestCalibs.py 

    Because the data ids are parsed out of the header for calibrations produced
by the calibration products pipeline, this will not work for user
curated data.  This removes extra code used only for defects.
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    97f70e4 View commit details
    Browse the repository at this point in the history

  2. Remove calibType override 

    It turns out that getDestination doesn't respect the calibType argument anyway,
so this option has been removed.
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    8bcc79a View commit details
    Browse the repository at this point in the history

  3. First go at ingesting defects as real butler datasets 

    Note I had to call them "deflects" so the mapper special casing didn't
shadow my attempts.

Also, this uses a put to persist the defects to avoid an intermediate file.
This is not the standard practice as usually calibs are on disk and are simply
linked into a data repository.
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    db6551a View commit details
    Browse the repository at this point in the history

  4. Make move mode for ingest work across devices 

    os.rename only works within a single physical drive.  Ths shutil.move
function will copy and then delete the remaining file.
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    30bf333 View commit details
    Browse the repository at this point in the history

  5. Support ingesting defects using ingestCalibs

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    162744b View commit details
    Browse the repository at this point in the history

  6. Ignore ingestDefects.py in the bin directory

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    9cf8547 View commit details
    Browse the repository at this point in the history

  7. Update tests for change in bad columns. 

    The binary data in obs_base were updated several years ago, but the
defects were not updated to agree.  Now that those defects have been
updated, the nominal values in the test must change as well.
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    7821591 View commit details
    Browse the repository at this point in the history

  8. Don't update validity range if not ingesting 

    I believe this and the next commit fix DM-16817
    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    89e29a4 View commit details
    Browse the repository at this point in the history

  9. Delegate table existence to CreateTable

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    e76587e View commit details
    Browse the repository at this point in the history

  10. Make dependency on meas_alg required

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    50ce2dd View commit details
    Browse the repository at this point in the history

  11. Move code to read defects to pipe_tasks from obs_base

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    7fcc77b View commit details
    Browse the repository at this point in the history

  12. Normalize path for read_defects

    @SimonKrughoff
    SimonKrughoff committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    9a280b8 View commit details
    Browse the repository at this point in the history