Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do ASR must specify the parameter “tgt_lang” ? (ASR 必须要指定tgt_lang这个参数吗) #305

Open
lilongwei5054 opened this issue Dec 25, 2023 · 1 comment

Comments

@lilongwei5054
Copy link

lilongwei5054 commented Dec 25, 2023

import torchaudio
from transformers import AutoProcessor, SeamlessM4Tv2Model
processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
model = SeamlessM4Tv2Model.from_pretrained("facebook/seamless-m4t-v2-large")

fileName="asr.wav"
audio, orig_freq = torchaudio.load(fileName)
audio = torchaudio.functional.resample(audio, orig_freq=orig_freq, new_freq=16000)
audio_inputs = processor(audios=audio, return_tensors="pt")
output_tokens = model.generate(audio_inputs, tgt_lang="cmn", generate_speech=False)
translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)
//ASR result : 今天天气真不错
When I set tgt_lang="cmn",the result is correct.The original audio is in Chinese.
But when I set tgt_lang=None,the ASR result is "The weather is really nice today".It has been translated into English!

I think it could have automatically determined the language in the audio

@lilongwei5054 lilongwei5054 changed the title Do ASR must specify the parameter “tgl_lang” ? (ASR 必须要指定tgt_lang这个参数吗) Do ASR must specify the parameter “tgt_lang” ? (ASR 必须要指定tgt_lang这个参数吗) Dec 25, 2023
@avidale
Copy link
Contributor

avidale commented Mar 14, 2024

ASR with Seamless is treated as a special case of translation, where the source and target languages are the same.
But the Seamless models were not trained to predict the target language on their own, so it is your responsibility to provide the right tgt_lang tag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants