Skip to content

Fix mistral checkpoint loading in utils/fetch_hub_objects_for_ci.py: avoid too many requests and/or timeout #43584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ydshieh merged 13 commits into from
Jan 29, 2026
Merged

Fix mistral checkpoint loading in utils/fetch_hub_objects_for_ci.py: avoid too many requests and/or timeout #43584

Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
6f15955
crazy mistral
ydshieh Jan 29, 2026
16eca69
trigger
ydshieh Jan 29, 2026
832fff6
how long I will remain crazy?
ydshieh Jan 29, 2026
2e529b9
call me The Phantom of the Mistral
ydshieh Jan 29, 2026
e19e5f0
I am The Phantom of the Mistral
ydshieh Jan 29, 2026
fe6e44c
les miserables
ydshieh Jan 29, 2026
69d500f
les miserables
ydshieh Jan 29, 2026
0ed6065
may I be in the peace now?
ydshieh Jan 29, 2026
00d9501
[test_all] please
ydshieh Jan 29, 2026
98410a2
wtf
ydshieh Jan 29, 2026
efa9a64
[test_all] please
ydshieh Jan 29, 2026
383b537
RIP
ydshieh Jan 29, 2026
735d5c2
[test_all] please
ydshieh Jan 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions utils/fetch_hub_objects_for_ci.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,20 +265,40 @@ def download_test_file(url):

# For `tests/test_tokenization_mistral_common.py:TestMistralCommonBackend`, which eventually calls
# `mistral_common.tokens.tokenizers.utils.download_tokenizer_from_hf_hub` which (probably) doesn't have the cache.
# For `revision=None`, see https://github.com/huggingface/transformers/pull/40623
if is_mistral_common_available():
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.tokens.tokenizers.utils import list_local_hf_repo_files

from transformers import AutoTokenizer
from transformers.tokenization_mistral_common import MistralCommonBackend

repo_id = "hf-internal-testing/namespace-mistralai-repo_name-Mistral-Small-3.1-24B-Instruct-2503"
AutoTokenizer.from_pretrained(repo_id, tokenizer_type="mistral")
MistralCommonBackend.from_pretrained(repo_id)
MistralTokenizer.from_hf_hub(repo_id)

# determine if we already have this downloaded
local_files_only = len(list_local_hf_repo_files(repo_id, revision=None)) > 0

# This will go the path `transformers/tokenization_mistral_common.py::MistralCommonBackend::from_pretrained --> mistral_common.tokens.tokenizers.utils.download_tokenizer_from_hf_hub`.
# No idea at all why we need the statement below again (`MistralCommonBackend.from_pretrained`).
AutoTokenizer.from_pretrained(
repo_id, tokenizer_type="mistral", local_files_only=local_files_only, revision=None
)

_ = MistralCommonBackend.from_pretrained(
repo_id,
local_files_only=local_files_only,
# This is a hack as `list_local_hf_repo_files` from `mistral_common` has a bug
# TODO: Discuss with `mistral-common` maintainers: after a fix being done there, remove this `revision` hack
revision=None,
)

MistralTokenizer.from_hf_hub(repo_id, local_files_only=local_files_only)

repo_id = "mistralai/Voxtral-Mini-3B-2507"
AutoTokenizer.from_pretrained(repo_id)
MistralTokenizer.from_hf_hub(repo_id)
local_files_only = len(list_local_hf_repo_files(repo_id, revision=None)) > 0

AutoTokenizer.from_pretrained(repo_id, local_files_only=local_files_only, revision=None)
MistralTokenizer.from_hf_hub(repo_id, local_files_only=local_files_only)

# Download files from URLs to local directory
for url in URLS_FOR_TESTING_DATA:
Expand Down