Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-38779: Use graph DatasetRef in execution butler #326

Merged
merged 6 commits into from
May 4, 2023
Merged

DM-38779: Use graph DatasetRef in execution butler #326

merged 6 commits into from
May 4, 2023

Commits on May 4, 2023

  1. Use graph DatasetRef in execution butler 

    This attempts to ensure we have provenance agreement between
the graph and the execution butler, since the graph now always
has resolved refs.
    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    299e396 View commit details
    Browse the repository at this point in the history

  2. No longer unresolve on butler put 

    A quantum graph always uses resolved refs and for provenance
we now always want to use that ref. Execution butler now records
datasets that match those in the graph so we can also now disable
the use of the special "put of predefined dataset" flag.

There are two caveats here:

* The code now assumes that the butler already knows about the
  dataset. Butler.put will not try to insert into registry.
* Can runQuantum be called with refs that butler knows nothing
  about.

There is the question of whether we can make this change until
unresolved refs are removed rather than deprecated.

In theory Butler.put() could be modified such that if the
dataset does not exist in registry (we do check for this)
we fall into the other code patch and try to add insert into
registry first.
    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    fdd8010 View commit details
    Browse the repository at this point in the history

  3. Check the provided run for execution butler creation

    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    0d5567d View commit details
    Browse the repository at this point in the history

  4. Add output run to graph metadata

    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    a577f0f View commit details
    Browse the repository at this point in the history

  5. Change test Utils code to return the registry refs 

    Unresolved refs are no longer allowed and the quantum should
always include a resolved ref now.
    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    66cb324 View commit details
    Browse the repository at this point in the history

  6. Add news fragment

    @timj
    timj committed May 4, 2023
    Configuration menu
    Copy the full SHA
    27a8167 View commit details
    Browse the repository at this point in the history