Mismatch between config.vocab_size and len(tokenizer) in Flan-T5

[Hannibal046]

System Info

• transformers version: 4.28.1
• Platform: Linux-5.15.0-1023-azure-x86_64-with-glibc2.17
• Python version: 3.8.16
• Huggingface_hub version: 0.14.1
• Safetensors version: not installed
• PyTorch version (GPU?): 1.13.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker and @younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer,AutoConfig
models = [
    "google/flan-t5-small",
    "google/flan-t5-base",
    "google/flan-t5-large",
    "google/flan-t5-xl",
    "google/flan-t5-xxl",
]
for model in models:
    config = AutoConfig.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    print(f"{model}\n\tlen(tokenizer)={len(tokenizer)},tokenizer.vocab_size={tokenizer.vocab_size},config.vocab_size={config.vocab_size}")

Expected behavior

The two are matched.

See #4875.

[Hannibal046]

Got it. Thanks! Voting for clarification.

I gather from the thread that it shouldn't be a problem. The size was increased for the ease of GPU usage. Is this creating any issues in inferencing or training the model? Just want to know for better understanding!

[Hannibal046]

Ok, here is my usecase.
I usually calculate loss myself rather than pass labels into a model for automatic loss calculation.
For example, when calculating commonly used Cross Entropy Loss, i would have to figure out vocab size myself.

loss = F.cross_entropy(
                        outputs.logits.view(-1,VOCAB_SIZE),
                        labels.view(-1),
                        label_smoothing=cfg.trainer.label_smoothing_factor
                        )

So i think in cases where vocab_size matters, the value from from config.vocab_size and len(tokenzier) should be consistent.

The pre-trained model which is provided by google had its vocab_size manually set to 32128 by them. Here's what I found in their github:

Did you try setting VOCAB_SIZE manually?

[Hannibal046]

Hi, thanks for the response! I just mean when i need to know the value of vocab_size, imo it should be consistent with len(tokenzier) and config.vocab_size in huggingface.