-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enh: Adding model package download functions to loads #55
Conversation
/blossom-ci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Users will indeed find this very useful, but I am not comfortable with the amount of I/O specific concerns in the packages, or with the backwards incompatible changes to Package.__init__
.
Package
should abstract such concerns and hide them from the load functions. I think a possible usage could be this:
package = Package("ngc://nvidia/modulus/modulus_dlwp_cubesphere/v0.1:zip")
earth2mip.networks.dlwp(package)
I'll have to think more about the entrypoint mechanism for defining model metadata. For CLI use, we already have a similar mechanism here:
earth2mip/earth2mip/_cli_utils.py
Line 21 in 5999d7e
def add_model_args(parser: argparse.ArgumentParser, required=False): |
/blossom-ci |
/blossom-ci |
/blossom-ci |
/blossom-ci |
/blossom-ci |
/blossom-ci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Just a couple minor comments, but I am hitting approve. Feel free to merge after addressing them. I don't think @yairchn needs to review this.
/blossom-ci |
Earth-2 MIP Pull Request
Description
Fixes: #54
This PR does a few items, however the primary objective is to enable auto-downloading of model packages that are known but are not present on the disk. This PR:
e2mip://
on the model nameearth2mip.networks
. This eliminates the requirement for metadata.json in favor of pure python. This is how the known models are discovered.Now running a ensemble config without a any of these models in the model registry just works.
Notebooks could be updated but they do provide good info about how these packages are set up. Should probably get rid of the sub-processes to wget commands to pythonic process. But one step at a time.
Open questions: How should we test this?
Checklist
Dependencies