-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support listing Hugging Face model info #4619
Conversation
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: smajumdar <smajumdar@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's extend list_available_models
method instead to accept optional ModelFilter instance. By default should be None and should list both NGC and HF models (even if duplicated). Then make list_available_models_on_hf
private and call it under the hood
Signed-off-by: smajumdar <smajumdar@nvidia.com>
I agree with this. We don't want users to have to use two different methods to list models. |
Reasons states belpw : it would require updating every single class that implements list_available_models which is impractical Problem is that method is subclassed by dozens of classes, and each of those subclasses need to be updated to support the new argument Another issue is that this method will fail if used in offline no internet area, whereas the original ngc ones has code baked urls, so it will still work. |
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Updated PR with different name for method to remove confusion against list_available_models |
Signed-off-by: smajumdar <smajumdar@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks for the changes!
* Support listing Hugging Face model info Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Update name of method, support list of model filters Signed-off-by: smajumdar <smajumdar@nvidia.com> * Improve docstring Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
* Support listing Hugging Face model info Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Update name of method, support list of model filters Signed-off-by: smajumdar <smajumdar@nvidia.com> * Improve docstring Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>
* Support listing Hugging Face model info Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Add documentation about usage Signed-off-by: smajumdar <smajumdar@nvidia.com> * Update name of method, support list of model filters Signed-off-by: smajumdar <smajumdar@nvidia.com> * Improve docstring Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
Signed-off-by: smajumdar smajumdar@nvidia.com
What does this PR do ?
Add new apis to filter and obtain NeMo hugging face models via code.
Collection: [Core]
Changelog
get_hf_model_filter()
andsearch_huggingface_models(model_filter=None)
to ModelPT.list_available_models_on_hf()
lists the nemo models that satisfy the filter.Usage
Before your PR is "Ready for review"
Pre checks:
PR Type: