How to combine timestamp prediction model in ASR pipeline?

[R1ckShi]

FunASR supports timestamp prediction for any ASR models by claiming inference_inference like:

The pipeline which only contains ASR model：

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

The pipeline which contains ASR and TP models (supported by FunASR >= 0.3.0)：

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    timestamp_model="damo/speech_timestamp_prediction-v1-16k-offline",
    timestamp_model_revision="v1.0.5",
)

rec_result = inference_pipeline(audio_in="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_timestamps.wav")
print(rec_result)

[R1ckShi]

For ASR model like uniasr or conformer, the inference pipelines do not support conduct timestamp prediction inside, you can use timestamp prediction pipeline outside the asr pipeline like:

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    )
inference_pipeline_tp = pipeline(
    task=Tasks.speech_timestamp,
    model='damo/speech_timestamp_prediction-v1-16k-offline',
    model_revision='v1.1.0',
    output_dir='./tmp')

rec_result_asr = inference_pipeline(audio_in="/Users/shixian/code/export/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav")
rec_result_tp = inference_pipeline_tp(audio_in="/Users/shixian/code/export/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav",
                                      text_in=" ".join(rec_result_asr['text']))
print(rec_result_asr, rec_result_tp)