Long translations end up switching back to English after 20 minutes

[chowder3907]

I am trying to use whisper to translate some longer videos (about an hour long ) to Spanish however after around the 20 minute mark, they end up not translating and only transcribing. After another ~20 minutes, it goes back to writing in Spanish. I'm not sure what I'm doing wrong. This is across multiple different videos and a consistent problem

[jongwook]

The models are not trained to transcribe into non-English languages, so it's actually not an intended use case (related: #649 (comment)). That being said, I guess in theory you could modify the decoding logic (the update() method below in particular`) to reject words or phrases in English.

whisper/whisper/decoding.py

Lines 199 to 211 in 0b5dcfd

	def update(self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor) -> Tuple[Tensor, bool]:
	"""Specify how to select the next token, based on the current trace and logits
	Parameters
	----------
	tokens : Tensor, shape = (n_batch, current_sequence_length)
	all tokens in the context so far, including the prefix and sot_sequence tokens
	logits : Tensor, shape = (n_batch, vocab_size)
	per-token logits of the probability distribution at the current step
	sum_logprobs : Tensor, shape = (n_batch)
	cumulative log probabilities for each sequence