Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-38454: butler mode for ModelPackages #21

Merged
merged 25 commits into from
Dec 15, 2023
Merged

DM-38454: butler mode for ModelPackages #21

merged 25 commits into from
Dec 15, 2023

Commits on Dec 12, 2023

  1. A sample script to test ingestion of pretrained models into 

    the Buler repo.
    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    6bae4b7 View commit details
    Browse the repository at this point in the history

  2. Pretrained models can now be stored into and loaded from butler, 

    with a custom formatter.
    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    011159e View commit details
    Browse the repository at this point in the history

  3. Ingest pretrained models under the pretrained_models collection. 

    E.g. the collection "pretrained_models/dummy" will be where the
pretrained weights for the model package "dummy" will be stored.
    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    0cfb290 View commit details
    Browse the repository at this point in the history

  4. Added the "butler-hybrid" mode to the ModelPackage interface. 

    This storage mode allows butler to load pretrained weights
from butler and pass them down to the task just like other
typical LSST tasks.
    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    e376b92 View commit details
    Browse the repository at this point in the history

  5. Move formatters to a separate module

    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    10daa85 View commit details
    Browse the repository at this point in the history

  6. Rename butler-hybrid to butler

    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    baaf5db View commit details
    Browse the repository at this point in the history

  7. Remove flake8 from pytest

    @NimSed
    NimSed committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    7bb791f View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. Decompose the import_model utility into its constituents. 

    This way we can also load a model/module which is already
loaded as a file-like object.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    3b11145 View commit details
    Browse the repository at this point in the history

  2. Add StorageAdapterButler to abstract out loading a ModelPackage 

    from a butler repository.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    c39824e View commit details
    Browse the repository at this point in the history

  3. (pre)load the whole package instead of just the weights.

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    06161ca View commit details
    Browse the repository at this point in the history

  4. Add the static ingest() method, as a convenience tool 

    for ingesting new model packages.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    cfa8567 View commit details
    Browse the repository at this point in the history

  5. Define sanity_check_dummy_model() to have a cleaner code.

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    6f375c2 View commit details
    Browse the repository at this point in the history

  6. Add TestModelPackageButler to test the butler-mode model packages 

    using a temporary (but real) butler repository.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    5c0334d View commit details
    Browse the repository at this point in the history

  7. Minor refactoring of tests. 

    Plus some minor fixes.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    6b20ca7 View commit details
    Browse the repository at this point in the history

  8. lookupFunctions are obselete now. Use optional connection instead.

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    ccc03d7 View commit details
    Browse the repository at this point in the history

  9. Major: define a more butler-oriented wrapper around modelPackages 

    to be passed back and forth between the formatter and
storageAdapterButler.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    75bf891 View commit details
    Browse the repository at this point in the history

  10. Prevent the user from specifying modelPackageName in the butler mode 

    (only for _real_ runs of pipelines)
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    1943e79 View commit details
    Browse the repository at this point in the history

  11. A demo script show-casing how to ingest a 'local' model-package into 

    butler.
    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    1676a07 View commit details
    Browse the repository at this point in the history

  12. Linting

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    974d22f View commit details
    Browse the repository at this point in the history

  13. Remove unused formatters

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    c8bc3d6 View commit details
    Browse the repository at this point in the history

  14. More linting 

    Co-authored-by: Jim Bosch <talljimbo@gmail.com>
    @NimSed @TallJimbo
    NimSed and TallJimbo committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    a556089 View commit details
    Browse the repository at this point in the history

  15. ~silence the exception 

    Co-authored-by: Jim Bosch <talljimbo@gmail.com>
    @NimSed @TallJimbo
    NimSed and TallJimbo committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    23b1488 View commit details
    Browse the repository at this point in the history

  16. Drop runQuantum's useless implementation :)

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    f3f6e60 View commit details
    Browse the repository at this point in the history

  17. Obselete the ButlerHybrid mode altogether.

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    5382a94 View commit details
    Browse the repository at this point in the history

  18. Add fromBytes(), toBytes() to NNModelPackageFormatter.

    @NimSed
    NimSed committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    ff1c4c5 View commit details
    Browse the repository at this point in the history