Parse safetensors metadata #1855
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
|
|
||
| To parse metadata from a single safetensors file, use [`get_safetensors_metadata`]. | ||
|
|
||
| For more details regarding the safetensors format, check out https://huggingface.co/docs/safetensors/index#format. |
There was a problem hiding this comment.
you'll be able to hyper-link to this implem from the safetensors doc too BTW (like i had done for the JS implem)
There was a problem hiding this comment.
more generally let's always make sure we cross-link stuff as much as possible
| ) | ||
| _headers = self._build_hf_headers(token=token) | ||
|
|
||
| # 1. Fetch first 100kb |
There was a problem hiding this comment.
Is this true? If I'm not mistaken @Narsil had told me 1MB but I didn't try it firsthand
There was a problem hiding this comment.
Narsil mentioned me 100kb as a good starting point.
Out of curiosity I've made a quick empirical study. I parsed 2700 files from the top 1000 models tagged as safetensors-compatible on the Hub, sorted by downloads. Out of 2700 files,
- maximum metadata header is 365kb
- 3.2% have a metadata header >=100kb
- 4.1% have a metadata header >=75kb
- 7.5% have a metadata header >= 50kb
- 18% have a metadata header >= 25kb
Given these numbers, 100kb looks like a good threshold. We could even lower it to 75kb but it's not worth it.
|
Thanks for the reviews @julien-c and @LysandreJik! I'll keep in mind to update the hub docs once this is released (#1855 (comment)) |
Related to #1832 cc @LysandreJik @Narsil
There are 2 methods:
parse_safetensors_file_metadata=> parses a safetensors file on the Hub => the "real" method that parses safetensorsget_safetensors_metadata=> takes a repo and parses all safetensors files (if sharded with amodel.safetensors.index.jsonfile) => more focused towardstransformersarchitecture (i.e. opinionated)I tried to follow more or less the typescript implementation with similar error handling.