Skip to content

Make tokenization_mistral_common importable without mistral_common installed - #47397

Merged
itazap merged 1 commit into
huggingface:mainfrom
juliendenize:fix-mistral-common-import-without-dependency
Jul 24, 2026
Merged

Make tokenization_mistral_common importable without mistral_common installed#47397
itazap merged 1 commit into
huggingface:mainfrom
juliendenize:fix-mistral-common-import-without-dependency

Conversation

@juliendenize

@juliendenize juliendenize commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

tokenization_mistral_common.py referenced mistral_common-only symbols at import time, so the module raised NameError when mistral_common was not installed. Those symbols are only imported inside the if is_mistral_common_available(): block, so they are undefined at module scope otherwise. This is not reachable through normal AutoTokenizer usage today since the module is only imported behind an is_mistral_common_available() guard, but this makes the module safe to import unconditionally.

The import-time references were the module-level _MAP_SPECIAL_TOKENS dict, the mode default arguments (ValidationMode.test, now the equivalent string "test"), and a few signature annotations referencing ValidationMode / ReasoningEffort (now quoted since the module has no from __future__ import annotations). Added a test that executes the module body with mistral_common both available and unavailable to ensure the import never raises.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline and the
    Pull Request checks?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes according to the guidelines?
  • Did you write any new necessary tests?

Who can review?

@ArthurZucker @itazap

Comment thread src/transformers/tokenization_mistral_common.py Outdated
@juliendenize
juliendenize force-pushed the fix-mistral-common-import-without-dependency branch from bb2eb64 to cf3c4d1 Compare July 20, 2026 12:16
@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 29640458203:2
Result: success | Jobs: 7 | Tests: 498 | Failures: 0 | Duration: 3m 50s

@Rocketknight1

Copy link
Copy Markdown
Member

looks ready for review so gentle ping @ArthurZucker @itazap

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@itazap itazap left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching!

@itazap
itazap added this pull request to the merge queue Jul 24, 2026
Merged via the queue into huggingface:main with commit 98a1e7d Jul 24, 2026
38 checks passed
stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026
Sainava pushed a commit to Sainava/Sai-transformers that referenced this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants