Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate output in list_models #1143

Merged
merged 6 commits into from
Nov 4, 2022

Conversation

Wauplin
Copy link
Contributor

@Wauplin Wauplin commented Oct 31, 2022

Fix #1087 and prepare future pagination of list_models, list_datasets and list_spaces (cc @Pierrci @julien-c - cf slack thread (internal link)).

The idea is to return a custom list on which any method that is specific to lists (e.g. almost everything except for item in list_models(): ...) is deprecated. This should encourage users to process the output as an iterator instead of a list. In particular, len(list_models()) is deprecated (as it will not be accurate anymore when the api will be paginated. Listing discussions from a repo is already paginated (python generator in hfh) which is good for consistency.

Implementation-wise, I'm using a metaclass to decorate all methods (so a bit python automagic) but I didn't see another way of doing it. For the record, I couldn't use the __getattribute__ method as it is not called for magic methods like __getitem__ or __len__.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 31, 2022

The documentation is not available anymore as the PR was closed or merged.

@Wauplin
Copy link
Contributor Author

Wauplin commented Nov 2, 2022

(Internal discussion here)

Note: there are 2 different topics here:

  1. Deprecate the list return value to prepare for the future iterator return value (which will iterate over pages)
  2. Handle pagination right now even if it will be implemented server side later (e.g. paginate over all pages, concatenate in a single list and output => suboptimal but future versions of hfh will not return a list anyway). Also possible that server-side we check the user-agent and don't paginate on older versions of hfh (=> would not break any existing library)

This PR currently handles 1. but not 2.

@codecov
Copy link

codecov bot commented Nov 3, 2022

Codecov Report

Base: 84.55% // Head: 84.69% // Increases project coverage by +0.13% 🎉

Coverage data is based on head (5a6f035) compared to base (91fe43c).
Patch coverage: 94.44% of modified lines in pull request are covered.

❗ Current head 5a6f035 differs from pull request most recent head cc4fe5f. Consider uploading reports for the commit cc4fe5f to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1143      +/-   ##
==========================================
+ Coverage   84.55%   84.69%   +0.13%     
==========================================
  Files          42       41       -1     
  Lines        4151     4130      -21     
==========================================
- Hits         3510     3498      -12     
+ Misses        641      632       -9     
Impacted Files Coverage Δ
src/huggingface_hub/utils/_deprecation.py 97.72% <93.75%> (-2.28%) ⬇️
src/huggingface_hub/hf_api.py 88.11% <100.00%> (+0.51%) ⬆️
src/huggingface_hub/repocard.py 93.78% <0.00%> (-0.27%) ⬇️
src/huggingface_hub/repocard_data.py 98.38% <0.00%> (-0.03%) ⬇️
src/huggingface_hub/utils/__init__.py 100.00% <0.00%> (ø)
src/huggingface_hub/utils/_subprocess.py 100.00% <0.00%> (ø)
src/huggingface_hub/utils/_git_credential.py
src/huggingface_hub/_login.py 52.00% <0.00%> (+4.94%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, works well! Thanks @Wauplin!

Comment on lines 191 to 196
assert (
"_deprecate" in attrs
), "A `_deprecate` method must be implemented to use `DeprecateListMetaclass`."
assert (
list in bases
), "Class must inherit from `list` to use `DeprecateListMetaclass`."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, I would favor raising errors with the appropriate types rather than assert, but if you feel like this is more coherent this way we can leave it be.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep true. I usually use assertions only for internal logic issues (like here where the metaclass is really not meant to be reused by anyone). But agree on changing this :)

@with_production_testing
def test_list_datasets(self):
def test_list_datasets_aaa(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this voluntary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahah, not a all 😄

@Wauplin Wauplin merged commit c2dbfd7 into main Nov 4, 2022
@Wauplin Wauplin deleted the 1087-deprecate-list-output-in-list-models branch November 4, 2022 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Warn users about future pagination on list_models / list_datasets / list_spaces
3 participants