Skip to content
This repository has been archived by the owner on Sep 13, 2023. It is now read-only.

Load models from GH #4

Closed
3 tasks
aguschin opened this issue Aug 30, 2021 · 3 comments
Closed
3 tasks

Load models from GH #4

aguschin opened this issue Aug 30, 2021 · 3 comments
Assignees
Labels
api MLEM Python API model registry Model registry functionality of MLEM
Milestone

Comments

@aguschin
Copy link
Contributor

When you do mlem.api.save(model, name), name can point to local fs, GitHub repo or other remote storages like s3, etc.
We need to

  • Create a precise textual description on how we should parse and resolve name
  • Write tests which describe this behaviour
  • Correct the actual code to satisfy tests
@aguschin aguschin self-assigned this Sep 10, 2021
@aguschin
Copy link
Contributor Author

Right now for load logic is the following: we have mlem.api.load function. Args include path, repo, rev.

  1. If you pass repo, then we suppose model is located in a repo.
  2. If you don't pass repo, then we suppose path is a fsspec-compatible path.

When this is decided, we first try to check whether we can just read the model by path, regardless of do we have .mlem folder somewhere or not. Path may be to a link file or model folder, it doesn't matter. If this fails, we assume that the path may be a link. We search for .mlem directory starting from the given path and going up, and when it is found, we treat all what's left as link name.

E.g., if we try to load("examples/dvc-pipeline/latest"), we will find examples/dvc-pipeline/.mlem folder and interpret latest as a link name. (We probably need to just interpret last part of URI as a model name, cause I don't think we want to support links having slash in names, like nn/latest).

Then we search .mlem directory for all objects with the same name (called latest in the example above) and raise an error if we found more or less than one. Otherwise we just load it.

@mike0sv, does this sound ok or something is missing?

@aguschin aguschin changed the title Refactor "name to actual path" resolution Load/save models to GH and fsspec paths Sep 14, 2021
@aguschin aguschin transferred this issue from another repository Sep 30, 2021
@aguschin aguschin added api MLEM Python API model registry Model registry functionality of MLEM labels Sep 30, 2021
@aguschin aguschin added this to the Alpha Release milestone Oct 7, 2021
@aguschin aguschin changed the title Load/save models to GH and fsspec paths Load models from GH Oct 12, 2021
@aguschin
Copy link
Contributor Author

In Alpha we've decided to address only "load" functionality.

"save to GH" functionality will be done in Beta in #18

@aguschin
Copy link
Contributor Author

As it works already, I close this ticket:

def test_model_loading_from_github(path):

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api MLEM Python API model registry Model registry functionality of MLEM
Projects
None yet
Development

No branches or pull requests

1 participant