Support expand parameter in xxx_info and list_xxxs (model/dataset/Space)
#2333
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Wauplin
changed the title
expand parameterin model_info and list_modelsexpand parameter in xxx_info and list_xxxs (model/dataset/Space)
Wauplin
requested review from
LysandreJik,
osanseviero and
julien-c
and removed request for
LysandreJik
julien-c
approved these changes
Jun 27, 2024
LysandreJik
approved these changes
Jul 11, 2024
|
Thanks for the reviews! 🤗 |
Wauplin
added a commit
that referenced
this pull request
Jul 11, 2024
* Use extended path on Windows when downloading to local dir Change the path of the local dir to an extended path by prepending "\\?\" to the absolute path, when the absolute path is longer than 255 characters on Windows. Also fixed a small typo. * Use extended path on Windows when downloading to local dir Change the path of the local dir to an extended path by prepending "\\?\" to the absolute path, when the absolute path is longer than 255 characters on Windows. Also fixed a small typo. * Move path handling to `get_local_download_paths()` for robustness On Windows we check the length of `lock_path` and if it is longer than 255 characters we prepend the `\\?\` prefix to all paths if it does not already exist. We only need to check the length of `lock_path` because it is guaranteed to be the longest path. * `safetensors[torch]` (#2371) * Fix token=False not respected in file download (#2386) * Fix token=False not respected in file download * lint * Handle shared layers in `save_torch_state_dict` + add `save_torch_model` (#2373) * Handle shared layers in save_torch_state_dict + save_torch_model + some helpers * fix pytest rerun * more reruns * Support `expand` parameter in `xxx_info` and `list_xxxs` (model/dataset/Space) (#2333) * First draft to support `expand` parameter for models * add expand support for dataset * add expand support for Space * Use extended path on Windows when downloading to local dir Change the path of the local dir to an extended path by prepending "\\?\" to the absolute path, when the absolute path is longer than 255 characters on Windows. Also fixed a small typo. * Move path handling to `get_local_download_paths()` for robustness On Windows we check the length of `lock_path` and if it is longer than 255 characters we prepend the `\\?\` prefix to all paths if it does not already exist. We only need to check the length of `lock_path` because it is guaranteed to be the longest path. * Use extended path on Windows when downloading to local dir Change the path of the local dir to an extended path by prepending "\\?\" to the absolute path, when the absolute path is longer than 255 characters on Windows. Also fixed a small typo. * Removed old path handling * Reorder path check; add tests * Skip test if opn Windows The test now shows up a `skipped` if executed on a non-Windows machine Co-authored-by: Lucain <lucainp@gmail.com> * Fix indentation for test_local_folder.py * Fix code style --------- Co-authored-by: Lucain <lucain@huggingface.co> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Lucain <lucainp@gmail.com>
julien-c
reviewed
Jul 17, 2024
| "downloads", | ||
| "downloadsAllTime", | ||
| "gated", | ||
| "gitalyUid", |
There was a problem hiding this comment.
hmm this is internal only i don't think any user will ever need it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Solve #2325.
EDIT: added support for
model_info/dataset_info/space_infoas well aslist_models/list_datasets/list_spaces. Added tests and docstrings for all of them.First draft of what we discussed in #2325 (comment) cc @osanseviero @julien-c . In the end I went for "put everything in
ModelInfoand have all attributes optional". For now I only implementedexpandfor models helpers but it can be easily extended to datasets/spaces.In order to better document the available parameters, I used a
Literaltype annotations listing all valid keys. Since it's "just" a type annotation, passing any string will still be backward compatible in the future as long as the server handles it. The cons is that the list has to be manually maintained to stay up to date with server changes (not happening very often). I have added a test for that. At least an update is pushed server-side, we will notice it in our CI runs.@osanseviero @julien-c let me know if you are opinionated on something, otherwise I'll continue with it :)