Error when launching Magistral-Small-2506

[RonaldGalea]

System Info

Issue seems to be specific to the model Magistral-Small-2506, as e.g. Mistral-Small-24B-Instruct-2501 works just fine with the exact same setup.

Error:

OSError: Can't load tokenizer for 'mistralai/Magistral-Small-2506'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'mistralai/Magistral-Small-2506' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
2025-07-04T15:32:32.089880Z ERROR text_generation_launcher: Error when initializing model

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

I am running TGI via docker compose, code below:

  tgi_app:
    image: "ghcr.io/huggingface/text-generation-inference:3.3.4"
    environment:
      - HF_TOKEN=asd
      - HF_HOME=/data/cache
      - HUGGINGFACE_HUB_CACHE=/data/cache/hub
      - HF_DATASETS_CACHE=/data/cache/datasets
    shm_size: 1g
    ports:
      - "8080:80"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - "/data:/data"
    command:
      - --model-id mistralai/Magistral-Small-2506
      - --usage-stats=off
      - --sharded true
      - --num-shard 4

Expected behavior

Would expect model to run without issues.

[AmazingGabriel16]

They won't release the tokenizer, use mistral small 3 2503's instead.

mistralai/Mistral-Small-3.1-24B-Instruct-2503