Fix AttributeError: MixtralModel object has no attribute org_vocab_size.

[Pernekhan]

Mixtral cannot start because of this error:

  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'MixtralModel' object has no attribute 'org_vocab_size'. Did you mean: 'vocab_size'?

Related PR: #2831

[pcmoritz]

You are right, this is indeed broken at the moment!

There seems to be some divergence on how things are working for Mistral vs. Mixtral (Mixtral has self.vocab_size = config.vocab_size whereas Mistral has self.vocab_size = config.vocab_size + lora_vocab and self.org_vocab_size = config.vocab_size) -- any thoughts about this @Yard1 ?

[Yard1]

For this, Mistral has the correct logic and Mixtral should follow it exactly.

[pcmoritz]

Let me try to update that @Pernekhan do you mind if I push to your PR to update it?

[pcmoritz]

Unfortunately I couldn't push to this PR, so made a new one including your commit @Pernekhan :)

#2880