can f5-tts inference more faster?

[yumoqing]

Can f5-tts inference more faster?

environment:

Machine type:vps
python:12.3
OS:ubuntu 22.04
MEM:64G
vCPU:16
GPU: one of nvidia 4090
install: follow the installation step in README.md

question:

the currently version of f5-tts use or not use GPU(by model_obj = model_obj.to('cuda:0')) run the inference examples in README.md need about 3 ~ 5 seconds to finish on the above environment, my question is: can f5-tts finish inference time less then 0.2 seconds. if ca, how to?

test code

the appPublic models can be git clone from here, after clone, go into the folder, run:

pip install -r requirements.txt
pip install .

import sys
sys.path.append('./F5TTS')
import argparse
import codecs
import re
from pathlib import Path

import numpy as np
import soundfile as sf
import tomli
from cached_path import cached_path

from model import DiT, UNetT
from model.utils_infer import (
    load_vocoder,
    load_model,
    preprocess_ref_audio_text,
    infer_process,
    remove_silence_for_generated_wav,
)

import os
import json
from time import time
from appPublic.dictObject import DictObject
from appPublic.zmq_reqrep import ZmqReplier
from appPublic.folderUtils import temp_file
from appPublic.jsonConfig import getConfig

n_mel_channels = 100
hop_length = 256
target_rms = 0.1
nfe_step = 32  # 16, 32
cfg_strength = 2.0
ode_method = "euler"
sway_sampling_coef = -1.0
speed = 1.0

class F5TTS:
    def __init__(self):
        self.config = getConfig()
        self.zmq_url = self.config.zmq_url
        self.replier = ZmqReplier(self.config.zmq_url, self.generate)
        # self.vocos = load_vocoder(is_local=True, local_path="../checkpoints/charactr/vocos-mel-24khz")
        self.load_model()
        self.setup_voice()

    def run(self):
        print(f'running {self.zmq_url}')
        self.replier._run()
        print('ended ...')

    def load_model(self):
        # load models
        ckpt_file = ''
        if self.config.modelname == "F5-TTS":
            model_cls = DiT
            model_cfg = dict(dim=1024, depth=22, heads=16, ff_mult=2, text_dim=512, conv_layers=4)
            if ckpt_file == "":
                repo_name = "F5-TTS"
                exp_name = "F5TTS_Base"
                ckpt_step = 1200000
                ckpt_file = str(cached_path(f"hf://SWivid/{repo_name}/{exp_name}/model_{ckpt_step}.safetensors"))

        elif self.config.modelname == "E2-TTS":
            model_cls = UNetT
            model_cfg = dict(dim=1024, depth=24, heads=16, ff_mult=4)
            if ckpt_file == "":
                repo_name = "E2-TTS"
                exp_name = "E2TTS_Base"
                ckpt_step = 1200000
                ckpt_file = str(cached_path(f"hf://SWivid/{repo_name}/{exp_name}/model_{ckpt_step}.safetensors"))

        self.model = load_model(model_cls, model_cfg, ckpt_file,
                        self.config.vocab_file)
        self.model = self.model.to(self.config.device)

    def generate(self, d):
        msg= d.decode('utf-8')
        data = DictObject(**json.loads(msg))
        print(data)
        t1 = time()
        for wav in self.inference(data.prompt, stream=data.stream):
            if data.stream:
                d = {
                    "reqid":data.reqid,
                    "b64wave":b64str(wav),
                    "finish":False
                }
                self.replier.send(json.dumps(d))
            else:
                t2 = time()
                d = {
                    "reqid":data.reqid,
                    "audio_file":wav,
                    "time_cost":t2 - t1
                }
                print(f'{d}')
                return json.dumps(d)
        t2 = time()
        d = {
            "reqid":data.reqid,
            "time_cost":t2 - t1,
            "finish":True
        }
        return json.dumps(d)

    def setup_voice(self):
        main_voice = {"ref_audio": self.config.ref_audio_fn,
                    "ref_text": self.config.ref_text}
        if "voices" not in self.config:
            voices = {"main": main_voice}
        else:
            voices = self.config["voices"]
            voices["main"] = main_voice
        for voice in voices:
            voices[voice]["ref_audio"], voices[voice]["ref_text"] = preprocess_ref_audio_text(
                voices[voice]["ref_audio"], voices[voice]["ref_text"]
            )
            print("Voice:", voice)
            print("Ref_audio:", voices[voice]["ref_audio"])
            print("Ref_text:", voices[voice]["ref_text"])
        self.voices = voices


    def inference(self, prompt):
        text_gen = prompt
        remove_silence = False
        generated_audio_segments = []
        reg1 = r"(?=\[\w+\])"
        chunks = re.split(reg1, text_gen)
        reg2 = r"\[(\w+)\]"
        for text in chunks:
            match = re.match(reg2, text)
            if match:
                voice = match[1]
            else:
                print("No voice tag found, using main.")
                voice = "main"
            if voice not in self.voices:
                print(f"Voice {voice} not found, using main.")
                voice = "main"
            text = re.sub(reg2, "", text)
            gen_text = text.strip()
            ref_audio = self.voices[voice]["ref_audio"]
            ref_text = self.voices[voice]["ref_text"]
            print(f"Voice: {voice}, {self.model}")
            audio, final_sample_rate, spectragram = \
                    infer_process(ref_audio, ref_text, gen_text, self.model)
            generated_audio_segments.append(audio)

        if generated_audio_segments:
            final_wave = np.concatenate(generated_audio_segments)
            fn = temp_file(suffix='.wav')
            with open(fn, "wb") as f:
                sf.write(f.name, final_wave, final_sample_rate)
                # Remove silence
                if remove_silence:
                    remove_silence_for_generated_wav(f.name)
            return fn


if __name__ == '__main__':
    # workdir = os.getcwd()
    # config = getConfig(workdir)
    tts = F5TTS()
    print('here')
    # tts.run()
    while True:
        print('prompt:')
        p = input()
        if p != '':
            t1 = time()
            f = tts.inference(p)
            t2 = time()
            print(f'{f}, cost {t2-t1} seconds')

test result:

突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道，我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？

---------------------------------------------------------------------------
gen_text 0 突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道，
gen_text 1 我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？
Generating audio in 2 batches...
  0%|                                                                                     | 0/2 [00:00<?, ?it/s]Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.685 seconds.
Prefix dict has been built successfully.
/d/ymq/ve/f5tts/lib/python3.12/site-packages/torch/functional.py:665: UserWarning: ComplexHalf support is experimental and many operators don't support it yet. (Triggered internally at ../aten/src/ATen/EmptyTensor.cpp:41.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
100%|█████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.21s/it]
/tmp/tmpudccuuky.wav, cost 4.455051898956299 seconds
------------------------------------first time-----------------------------------------
gen_text 0 突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道，
gen_text 1 我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？
Generating audio in 2 batches...
100%|█████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.72s/it]
/tmp/tmpe0elbnfq.wav, cost 3.469146966934204 seconds
------------------------------------secondth time-----------------------------------------
gen_text 0 突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道，
gen_text 1 我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？
Generating audio in 2 batches...
100%|█████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.67s/it]
/tmp/tmpjzrnoi3a.wav, cost 3.3618202209472656 secondstest code
------------------------------------third time-----------------------------------------

Many thanks

[yumoqing]

config file

{
    "zmq_url" : "tcp://127.0.0.1:10003",
    "sample_rate":16000,
    "vocab_file":"",
    "remove_silence":false,
    "modelname":"F5-TTS",
    "device":"cuda:0",
    "ref_audio_fn":"$[workdir]$/samples/test_zh_1_ref_short.wav",
    "ref_text":"对，这就是我，万人敬仰的太乙真人。",
    "cross_fade_duration":0
}

[SWivid]

Possible solution:

1. Use less nfe_step for speed-quality trade-off.
2. Try distillation techniques.
3. Train a smaller model from scratch if single-language needed application scenario.

0.2 seconds for generating a ~20 seconds speech is with a RTF of 0.01, thought no existing TTS model could do that,
and do we really need a model with RTF=0.01?

[yumoqing]

Thanks for your reply, I'm trying to build a realtime oral-to-oral llm chat app, it will integrates asr + llm + tts, vad is also need for reaction in sentences level, maybe a tts with streamming inference will let users feels better.

I will try f5-tts again, use your No. 1 suggested solution, and check if the inference speed and audio quality.

[SWivid]

oral-to-oral llm chat app

Yes, in this way, a streamable tts is better √

[ehsanahmadkhan525]

Looking for the same solution can we implement streaming on F5? Please guide

[sonicnerd14]

Thanks for your reply, I'm trying to build a realtime oral-to-oral llm chat app, it will integrates asr + llm + tts, vad is also need for reaction in sentences level, maybe a tts with streamming inference will let users feels better.

I will try f5-tts again, use your No. 1 suggested solution, and check if the inference speed and audio quality.

I'm looking to do something similar, or really building upon an existing open-source repository. One important question I have is how much VRAM would your GPU need to get such quick inference speeds? Will the quality of the audio be reduced in turn for a quicker output time? Currently, it can take several minutes to generate just a 10-30s clip with vanilla F5 on an RTX 3070. So, without some type of fine-tuning how could you get blazing fast streaming out of this model?

[zhli2]

Can this model support streaming model or give me some modify suggestion. thx.

[yumoqing]

oral-to-oral llm chat app

Yes, in this way, a streamable tts is better √

F5-TTS update very quickly, compare to one or two week ago version, the inference-cli.py source code is more simple the day before yestday, and now disappear.

F5-TTS should be installable as a python module, and provide a stable API for user, I think it is very import to let user use the same API after F5TTS upgraded.
The following is just a suggestion
when Installation

pip install F5TTS
or
pip install git+https://github.com/SWivid/F5-TTS.git

in app code

from F5TTS import load_model
model = load_model(....)
"""
infernece() return a audio_file_name
"""
model.inference(ref_audio, ref_text, gen_text, **kwargs)
"""
inference_stream() return numpy.ndarray or wave data binary array
"""
model.inference_stream(ref_audio, ref_text, gen_text, **kwargs)

model.close()

[SWivid]

@yumoqing Hi, the inference-cli.py is still there~
f5-tts_infer-cli is exactly the previous inference-cli.py. When you do pip install -e ., you just get a more convenient way of using with the shortcut f5-tts_infer-cli

F5-TTS/pyproject.toml

Line 58 in e963929

"f5-tts_infer-cli" = "f5_tts.infer.infer_cli:main"

Check here,
https://github.com/SWivid/F5-TTS?tab=readme-ov-file#3-more-instructions
it gives the instruction, detailed guidance pointing to the current infer directory

where src/f5_tts/infer/infer_cli.py is just the previous inference-cli.py

[SWivid]

will close this issue, feel free to open if further questions|
welcome pr for any possible enhancement~