Support of the XTTS 2 in tts-server

[bivashy]

Related issues: #3454 #3271 #3140

---

Description

This Pull Request changes behaviour of Xtts#load_checkpoint, and ModelManager#download_model.
Changes:

• Excluded xtts 2 from configuration ignore list in ModelManager for proper config_path resolution in tts-server
• Swapped checkpoint_path, checkpoint_dir parameters to follow the BaseTrainerModel implementation

Fixes:

Traceback (most recent call last):
  File "/root/TTS/server/server.py", line 104, in <module>
    synthesizer = Synthesizer(
  File "/root/TTS/utils/synthesizer.py", line 93, in __init__
    self._load_tts(tts_checkpoint, tts_config_path, use_cuda)
  File "/root/TTS/utils/synthesizer.py", line 183, in _load_tts
    self.tts_config = load_config(tts_config_path)
  File "/root/TTS/config/__init__.py", line 82, in load_config
    ext = os.path.splitext(config_path)[1]
  File "/usr/lib/python3.10/posixpath.py", line 118, in splitext
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

---

Current state

This Pull Request is tested with xtts2 model, and without specifying xtts2 model, it works fine.
Have a nice day!

[ther3zz]

@bivashy I tried your changes on my end with a fine tuned model (xtts v2) and I could not get it to work. Were you successful using a fine tuned model?

I'm running it like:
tts-server --model_path /models/test_v2/ --config_path /models/test_v2/config.json --use_cuda true

And receive the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/root/TTS/tts/models/../.models.json'

edit: full error:
Traceback (most recent call last): File "/usr/local/bin/tts-server", line 33, in <module> sys.exit(load_entry_point('TTS', 'console_scripts', 'tts-server')()) File "/usr/local/bin/tts-server", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load module = import_module(match.group('module')) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/root/TTS/server/server.py", line 104, in <module> synthesizer = Synthesizer( File "/root/TTS/utils/synthesizer.py", line 93, in __init__ self._load_tts(tts_checkpoint, tts_config_path, use_cuda) File "/root/TTS/utils/synthesizer.py", line 183, in _load_tts self.tts_config = load_config(tts_config_path) File "/root/TTS/config/__init__.py", line 97, in load_config config_class = register_config(model_name.lower()) File "/root/TTS/config/__init__.py", line 40, in register_config from TTS.tts.configs.xtts_config import XttsConfig File "/root/TTS/tts/configs/xtts_config.py", line 5, in <module> from TTS.tts.models.xtts import XttsArgs, XttsAudioConfig File "/root/TTS/tts/models/xtts.py", line 67, in <module> manager = ModelManager(path) File "/root/TTS/utils/manage.py", line 56, in __init__ self.read_models_file(models_file) File "/root/TTS/utils/manage.py", line 68, in read_models_file self.models_dict = read_json_with_comments(file_path) File "/root/TTS/config/__init__.py", line 17, in read_json_with_comments with fsspec.open(json_path, "r", encoding="utf-8") as f: File "/usr/local/lib/python3.10/dist-packages/fsspec/core.py", line 103, in __enter__ f = self.fs.open(self.path, mode=mode) File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 1295, in open f = self._open( File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 180, in _open return LocalFileOpener(path, mode, fs=self, **kwargs) File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 302, in __init__ self._open() File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 307, in _open self.f = open(self.path, mode=self.mode) FileNotFoundError: [Errno 2] No such file or directory: '/root/TTS/tts/models/../.models.json'

[bivashy]

@bivashy I tried your changes on my end with a fine tuned model (xtts v2) and I could not get it to work. Were you successful using a fine tuned model?

I'm running it like:
tts-server --model_path /models/test_v2/ --config_path /models/test_v2/config.json --use_cuda true

And receive the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/root/TTS/tts/models/../.models.json'

edit: full error:
Traceback (most recent call last): File "/usr/local/bin/tts-server", line 33, in <module> sys.exit(load_entry_point('TTS', 'console_scripts', 'tts-server')()) File "/usr/local/bin/tts-server", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load module = import_module(match.group('module')) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/root/TTS/server/server.py", line 104, in <module> synthesizer = Synthesizer( File "/root/TTS/utils/synthesizer.py", line 93, in __init__ self._load_tts(tts_checkpoint, tts_config_path, use_cuda) File "/root/TTS/utils/synthesizer.py", line 183, in _load_tts self.tts_config = load_config(tts_config_path) File "/root/TTS/config/__init__.py", line 97, in load_config config_class = register_config(model_name.lower()) File "/root/TTS/config/__init__.py", line 40, in register_config from TTS.tts.configs.xtts_config import XttsConfig File "/root/TTS/tts/configs/xtts_config.py", line 5, in <module> from TTS.tts.models.xtts import XttsArgs, XttsAudioConfig File "/root/TTS/tts/models/xtts.py", line 67, in <module> manager = ModelManager(path) File "/root/TTS/utils/manage.py", line 56, in __init__ self.read_models_file(models_file) File "/root/TTS/utils/manage.py", line 68, in read_models_file self.models_dict = read_json_with_comments(file_path) File "/root/TTS/config/__init__.py", line 17, in read_json_with_comments with fsspec.open(json_path, "r", encoding="utf-8") as f: File "/usr/local/lib/python3.10/dist-packages/fsspec/core.py", line 103, in __enter__ f = self.fs.open(self.path, mode=mode) File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 1295, in open f = self._open( File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 180, in _open return LocalFileOpener(path, mode, fs=self, **kwargs) File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 302, in __init__ self._open() File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 307, in _open self.f = open(self.path, mode=self.mode) FileNotFoundError: [Errno 2] No such file or directory: '/root/TTS/tts/models/../.models.json'

Hello, i am sorry to hear that. I have tested it only with model_path, but without config_path
I'll try to fix that when I would've have

UPD: I will try to solve the problem saturday and sunday

[bivashy]

I'm running it like: tts-server --model_path /models/test_v2/ --config_path /models/test_v2/config.json --use_cuda true

Try to run like this:
tts-server --model_path /models/test_v2/model.pth --config_path /models/test_v2/config.json --use_cuda true
Because I couldn't find in the documentation where it says that model_path can be a directory.

[ther3zz]

I'm running it like: tts-server --model_path /models/test_v2/ --config_path /models/test_v2/config.json --use_cuda true

Try to run like this: tts-server --model_path /models/test_v2/model.pth --config_path /models/test_v2/config.json --use_cuda true Because I couldn't find in the documentation where it says that model_path can be a directory.

Hey @bivashy , just tried that way and I receive the following error:
Traceback (most recent call last): File "/usr/local/bin/tts-server", line 33, in <module> sys.exit(load_entry_point('TTS', 'console_scripts', 'tts-server')()) File "/usr/local/bin/tts-server", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load module = import_module(match.group('module')) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/root/TTS/server/server.py", line 104, in <module> synthesizer = Synthesizer( File "/root/TTS/utils/synthesizer.py", line 93, in __init__ self._load_tts(tts_checkpoint, tts_config_path, use_cuda) File "/root/TTS/utils/synthesizer.py", line 183, in _load_tts self.tts_config = load_config(tts_config_path) File "/root/TTS/config/__init__.py", line 97, in load_config config_class = register_config(model_name.lower()) File "/root/TTS/config/__init__.py", line 40, in register_config from TTS.tts.configs.xtts_config import XttsConfig File "/root/TTS/tts/configs/xtts_config.py", line 5, in <module> from TTS.tts.models.xtts import XttsArgs, XttsAudioConfig File "/root/TTS/tts/models/xtts.py", line 67, in <module> manager = ModelManager(path) File "/root/TTS/utils/manage.py", line 56, in __init__ self.read_models_file(models_file) File "/root/TTS/utils/manage.py", line 68, in read_models_file self.models_dict = read_json_with_comments(file_path) File "/root/TTS/config/__init__.py", line 17, in read_json_with_comments with fsspec.open(json_path, "r", encoding="utf-8") as f: File "/usr/local/lib/python3.10/dist-packages/fsspec/core.py", line 103, in __enter__ f = self.fs.open(self.path, mode=mode) File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 1295, in open f = self._open( File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 180, in _open return LocalFileOpener(path, mode, fs=self, **kwargs) File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 302, in __init__ self._open() File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 307, in _open self.f = open(self.path, mode=self.mode) FileNotFoundError: [Errno 2] No such file or directory: '/root/TTS/tts/models/../.models.json'

[bivashy]

I'm running it like: tts-server --model_path /models/test_v2/ --config_path /models/test_v2/config.json --use_cuda true

Try to run like this: tts-server --model_path /models/test_v2/model.pth --config_path /models/test_v2/config.json --use_cuda true Because I couldn't find in the documentation where it says that model_path can be a directory.

Hey @bivashy , just tried that way and I receive the following error: ...

Maybe we should move into the discussions?
#3544

[ther3zz]

Putting this here in case someone else gets stuck, I was able to get things working for my setup:
#3544

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[nonedef]

@bivashy I've tried what you've changed but still get errors

server.py

use_multi_speaker = (hasattr(synthesizer.tts_model, "num_speakers") and (
    synthesizer.tts_model.num_speakers > 1 or synthesizer.tts_speakers_file is not None
)) or (speaker_manager is not None)

speaker_manager = getattr(synthesizer.tts_model, "speaker_manager", None)

use_multi_language = (hasattr(synthesizer.tts_model, "num_languages") and (
    synthesizer.tts_model.num_languages > 1 or synthesizer.tts_languages_file is not None
)) or (language_manager is not None)

language_manager = getattr(synthesizer.tts_model, "language_manager", None)

 # TODO: set this from SpeakerManager
use_gst = synthesizer.tts_config.get("use_gst", False)
app = Flask(__name__)

xtts.py

      """
        Loads a checkpoint from disk and initializes the model's state and tokenizer.

        Args:
            config (dict): The configuration dictionary for the model.
            checkpoint_dir (str, optional): The directory where the checkpoint is stored. Defaults to None.
            checkpoint_path (str, optional): The path to the checkpoint file. Defaults to None.
            vocab_path (str, optional): The path to the vocabulary file. Defaults to None.
            eval (bool, optional): Whether to set the model to evaluation mode. Defaults to True.
            strict (bool, optional): Whether to strictly enforce that the keys in the checkpoint match the keys in the model. Defaults to True.

        Returns:
            None
        """
        if checkpoint_dir is None and checkpoint_path:
            checkpoint_dir = os.path.dirname(checkpoint_path)
            
        model_path = checkpoint_path or os.path.join(checkpoint_dir, "model.pth")
        vocab_path = vocab_path or os.path.join(checkpoint_dir, "vocab.json")

        if speaker_file_path is None and checkpoint_dir is not None:
            speaker_file_path = os.path.join(checkpoint_dir, "speakers_xtts.pth")

manage.py

       # find downloaded files
        output_model_path = output_path
        output_config_path = None
        if (
            model not in ["tortoise-v2", "bark"] and "fairseq" not in model_name and "xtts1" not in model_name
        ):  # TODO:This is stupid but don't care for now.
            output_model_path, output_config_path = self._find_files(output_path)
        # update paths in the config.json
        self._update_paths(output_path, output_config_path)
        return output_model_path, output_config_path, model_item

After running python3 server.py --model_name tts_models/multilingual/multi-dataset/xtts_v2

> tts_models/multilingual/multi-dataset/xtts_v2 is already downloaded.
> Using model: xtts

Traceback (most recent call last):
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\server\server.py", line 104, in <module>
   synthesizer = Synthesizer(
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\utils\synthesizer.py", line 93, in __init__
   self._load_tts(tts_checkpoint, tts_config_path, use_cuda)
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\utils\synthesizer.py", line 192, in _load_tts
   self.tts_model.load_checkpoint(self.tts_config, tts_checkpoint, eval=True)
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\tts\models\xtts.py", line 775, in load_checkpoint
   checkpoint = self.get_compatible_checkpoint_state_dict(model_path)
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\tts\models\xtts.py", line 714, in get_compatible_checkpoint_state_dict
   checkpoint = load_fsspec(model_path, map_location=torch.device("cpu"))["model"]
 File "H:\HP Z440\Documents\H\Adham\GitHub\TTS-dev\TTS\utils\io.py", line 46, in load_fsspec
   with fsspec.open(
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\core.py", line 103, in __enter__
   f = self.fs.open(self.path, mode=mode)
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\cached.py", line 449, in <lambda>
   return lambda *args, **kw: getattr(type(self), item).__get__(self)(
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\spec.py", line 1293, in open
   f = self._open(
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\cached.py", line 449, in <lambda>
   return lambda *args, **kw: getattr(type(self), item).__get__(self)(
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\cached.py", line 695, in _open
   fn = self._make_local_details(path)
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\cached.py", line 449, in <lambda>
   return lambda *args, **kw: getattr(type(self), item).__get__(self)(
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\cached.py", line 612, in _make_local_details
   "uid": self.fs.ukey(path),
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\spec.py", line 1332, in ukey
   return sha256(str(self.info(path)).encode()).hexdigest()
 File "C:\Users\HP Z440\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\implementations\local.py", line 92, in info
   out = os.stat(path, follow_symlinks=False)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:/Users/HP Z440/AppData/Local/tts/tts_models--multilingual--multi-dataset--xtts_v2/model.pth/model.pth