Something wrong with OpenAI's Large V2 model?

[ggerganov]

So, I haven't looked in details, but I suspect there might be something wrong in the new large model released by OpenAI. Keep in mind this is very anecdotal evidence atm, so I might be completely wrong.

Running the main example with enabled color coding for the token probabilities -pc we normally get the following results:

• base.en

• small.en

• small

• medium

• medium.en

However, this is what the color coding look like when using the new large model (i.e. v2):

• large

As a comparison, this is the same run, but using the old large model - i.e. v1:

• large-v1

So somehow the logits with v2 seem to be all over the place which is not observed for any of the other models.
Still, I need to double check all these observations, but I think there is something not quite right with large-v2

[Alumniminium]

Please stop calling it v2. There's an actual v2 now.
https://huggingface.co/openai/whisper-large-v2

[bayesian]

large_v2 is trained with BPE dropout, so its target tokens are a stochastic distribution. That might explain smoother logits from large_v2