WhisperX return translated output instead of normal transcription

[BankNatchapol]

I tried to use fine-tuned model with whisperx, so i first convert the model using this code.

import ctranslate2
from transformers import AutoTokenizer, AutoProcessor

# converting model to CTranslate2

model_path = "biodatlab/whisper-th-large-v3"
output_dir = ""

converter = ctranslate2.converters.TransformersConverter(
    model_name_or_path=model_path,
    load_as_float16=None
)

converter.convert(output_dir=output_dir, quantization="float16", force=True)
print(f"Model successfully converted to CTranslate2 format at {output_dir}")

then run transcribe

import whisperx
lang = "th"
device = 'cuda'

## WhisperX
batch_size_x = 8 # reduce if low on GPU mem
compute_type_x = "float16"

asr_options = {
      "max_new_tokens": None,
      "clip_timestamps": None,
      "hallucination_silence_threshold": None,

}
model_x = whisperx.load_model("", 
'cuda', 
compute_type=compute_type_x, 
language='th', asr_options=asr_options)

print(model_x.transcribe('test.wav', language='th', task='transcribe'))

The output should be in 'th' language, instead the output is mostly 'en'.

..., {'text': ' When you try, you will find a gap in between. When you miss, you will sit on the hard floor. It makes you want to get up and fight. But when you fight and you start to get something, you will find a gap. This gap is a trap. Some people are good at it, but they miss it. Some people are good at it, but they miss it. Some people are good at it, but they miss it. Some people are good at it, but they miss it. Some people are good at it, but they miss it.', 'start': 1120.026, 'end': 1140.435}, {'text': '       ', 'start': 1140.435, 'end': 1159.053}, {'text': '          ', 'start': 1159.053, 'end': 1185.282}, {'text': ' The best in Thailand, the first in the Olympic life, has been reading books all the time. He has won the Olympic gold medal.', 'start': 1185.282, 'end': 1203.08}], 'language': 'th'}

In the wav file, he's speaking in 'th', but somehow the transcription is the translation of his speech.

Anything to fix this?
Thank you in advance.

[sunarowicz]

Did you copied tokenizer.json and preprocessor_config.json from original to ct2 converted model? These two has to be included in the converted model directory.

I have the same experience (translation instead of transcription) with some other fine-tuned model where tokenizer.json is missing in the original model. When I used tokenizer.json from some other same language model, the audio is transcribed in correct language (not translated), but the quality is low (I guess because of the not fully compatible tokenizer taken from other model).

I use this command to convert to ct2:
ct2-transformers-converter --model <hf_model_name> --output_dir <dir_name> --copy_files tokenizer.json preprocessor_config.json

[edikici]

Hi! Do you have any updates on this issue?

I am running into the same problem: my model is fine-tuned based on OpenAI's whisper large-v3, and then converted into ct2 using ct2-transformers-converter. I copy *.json from the fine-tuned model directory into that of the converted one, but when I try to transcribe a set of audio recordings in French with the language='fr', task='transcribe' parameters, some (not all?!) of the transcripts are generated in English -- and they seem to be translations of what was said in French.
The only way to generate transcripts fully in French is to change the parameters to language='fr', task='translate', but then the accuracy of these transcriptions are way worse than that of the non-converted fine-tuned model.

Does anyone have a clue?
Thanks in advance!