Support listing Hugging Face model info

[titu1994]

Signed-off-by: smajumdar smajumdar@nvidia.com

What does this PR do ?

Add new apis to filter and obtain NeMo hugging face models via code.

Collection: [Core]

Changelog

• Adds two methods get_hf_model_filter() and search_huggingface_models(model_filter=None) to ModelPT.
• First method returns a basic ModelFilter object, can be overridden by the user. Used as default to next method if nothing is provided.
• Add support for metadata attributes to the filter to modify the result list of ModelInfo.
• list_available_models_on_hf() lists the nemo models that satisfy the filter.

Usage

from nemo.core import ModelPT

# You can replace <DomainSubclass> with any subclass of ModelPT.
# Get default ModelFilter
filt = <DomainSubclass>.get_hf_model_filter()

# Make any modifications to the filter as necessary
filt.language = [...]
filt.task = ...
filt.tags = [...]

# Add any metadata to the filter as needed
filt.limit_results = 5

# Obtain model info
model_infos = <DomainSubclass>.search_huggingface_models(model_filter=filt)

# Browse through cards and select an appropriate one
card = model_infos[0]

# Restore model using `modelId` of the card.
model = ModelPT.from_pretrained(card.modelId)

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

[okuchaiev]

Let's extend list_available_models method instead to accept optional ModelFilter instance. By default should be None and should list both NGC and HF models (even if duplicated). Then make list_available_models_on_hf private and call it under the hood

[ericharper]

Let's extend list_available_models method instead to accept optional ModelFilter instance. By default should be None and should list both NGC and HF models (even if duplicated). Then make list_available_models_on_hf private and call it under the hood

I agree with this. We don't want users to have to use two different methods to list models.

[titu1994]

Reasons states belpw : it would require updating every single class that implements list_available_models which is impractical

---

Problem is that method is subclassed by dozens of classes, and each of those subclasses need to be updated to support the new argument

Another issue is that this method will fail if used in offline no internet area, whereas the original ngc ones has code baked urls, so it will still work.

[titu1994]

Updated PR with different name for method to remove confusion against list_available_models

[ericharper]

LGTM. Thanks for the changes!