[Bug] Loading XTTS via Xtts.load_checkpoint()

[caffeinetoomuch]

Describe the bug

When loading the model using Xtts.load_checkpoint, exception is raised as Error(s) in loading state_dict for Xtts, which leads to missing keys GPT embedding weights and size mismatch on Mel embedding. Even tried providing the directory which had base(v2) model checkpoints and got the same result.

To Reproduce

import os
import torch
import torchaudio
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

print("Loading model...")
config = XttsConfig()
config.load_json("/path/to/xtts/config.json")
model = Xtts.init_from_config(config)
model.load_checkpoint(config, checkpoint_dir="/path/to/xtts/", use_deepspeed=True)
model.cuda()

print("Computing speaker latents...")
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(audio_path=["reference.wav"])

print("Inference...")
out = model.inference(
    "It took me quite a long time to develop a voice and now that I have it I am not going to be silent.",
    "en",
    gpt_cond_latent,
    speaker_embedding,
    temperature=0.7, # Add custom parameters here
)
torchaudio.save("xtts.wav", torch.tensor(out["wav"]).unsqueeze(0), 24000)

Expected behavior

Load the checkpoint and run inference without exception.

Logs

11-08 22:13:53 [__main__    ] ERROR    - Error(s) in loading state_dict for Xtts:
	Missing key(s) in state_dict: "gpt.gpt.wte.weight", "gpt.prompt_embedding.weight", "gpt.prompt_pos_embedding.emb.weight", "gpt.gpt_inference.transformer.h.0.ln_1.weight", "gpt.gpt_inference.transformer.h.0.ln_1.bias", "gpt.gpt_inference.transformer.h.0.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.0.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.0.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.0.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.0.ln_2.weight", "gpt.gpt_inference.transformer.h.0.ln_2.bias", "gpt.gpt_inference.transformer.h.0.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.0.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.0.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.0.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.1.ln_1.weight", "gpt.gpt_inference.transformer.h.1.ln_1.bias", "gpt.gpt_inference.transformer.h.1.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.1.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.1.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.1.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.1.ln_2.weight", "gpt.gpt_inference.transformer.h.1.ln_2.bias", "gpt.gpt_inference.transformer.h.1.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.1.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.1.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.1.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.2.ln_1.weight", "gpt.gpt_inference.transformer.h.2.ln_1.bias", "gpt.gpt_inference.transformer.h.2.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.2.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.2.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.2.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.2.ln_2.weight", "gpt.gpt_inference.transformer.h.2.ln_2.bias", "gpt.gpt_inference.transformer.h.2.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.2.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.2.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.2.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.3.ln_1.weight", "gpt.gpt_inference.transformer.h.3.ln_1.bias", "gpt.gpt_inference.transformer.h.3.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.3.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.3.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.3.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.3.ln_2.weight", "gpt.gpt_inference.transformer.h.3.ln_2.bias", "gpt.gpt_inference.transformer.h.3.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.3.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.3.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.3.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.4.ln_1.weight", "gpt.gpt_inference.transformer.h.4.ln_1.bias", "gpt.gpt_inference.transformer.h.4.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.4.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.4.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.4.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.4.ln_2.weight", "gpt.gpt_inference.transformer.h.4.ln_2.bias", "gpt.gpt_inference.transformer.h.4.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.4.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.4.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.4.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.5.ln_1.weight", "gpt.gpt_inference.transformer.h.5.ln_1.bias", "gpt.gpt_inference.transformer.h.5.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.5.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.5.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.5.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.5.ln_2.weight", "gpt.gpt_inference.transformer.h.5.ln_2.bias", "gpt.gpt_inference.transformer.h.5.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.5.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.5.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.5.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.6.ln_1.weight", "gpt.gpt_inference.transformer.h.6.ln_1.bias", "gpt.gpt_inference.transformer.h.6.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.6.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.6.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.6.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.6.ln_2.weight", "gpt.gpt_inference.transformer.h.6.ln_2.bias", "gpt.gpt_inference.transformer.h.6.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.6.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.6.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.6.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.7.ln_1.weight", "gpt.gpt_inference.transformer.h.7.ln_1.bias", "gpt.gpt_inference.transformer.h.7.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.7.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.7.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.7.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.7.ln_2.weight", "gpt.gpt_inference.transformer.h.7.ln_2.bias", "gpt.gpt_inference.transformer.h.7.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.7.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.7.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.7.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.8.ln_1.weight", "gpt.gpt_inference.transformer.h.8.ln_1.bias", "gpt.gpt_inference.transformer.h.8.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.8.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.8.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.8.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.8.ln_2.weight", "gpt.gpt_inference.transformer.h.8.ln_2.bias", "gpt.gpt_inference.transformer.h.8.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.8.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.8.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.8.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.9.ln_1.weight", "gpt.gpt_inference.transformer.h.9.ln_1.bias", "gpt.gpt_inference.transformer.h.9.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.9.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.9.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.9.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.9.ln_2.weight", "gpt.gpt_inference.transformer.h.9.ln_2.bias", "gpt.gpt_inference.transformer.h.9.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.9.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.9.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.9.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.10.ln_1.weight", "gpt.gpt_inference.transformer.h.10.ln_1.bias", "gpt.gpt_inference.transformer.h.10.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.10.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.10.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.10.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.10.ln_2.weight", "gpt.gpt_inference.transformer.h.10.ln_2.bias", "gpt.gpt_inference.transformer.h.10.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.10.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.10.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.10.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.11.ln_1.weight", "gpt.gpt_inference.transformer.h.11.ln_1.bias", "gpt.gpt_inference.transformer.h.11.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.11.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.11.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.11.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.11.ln_2.weight", "gpt.gpt_inference.transformer.h.11.ln_2.bias", "gpt.gpt_inference.transformer.h.11.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.11.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.11.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.11.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.12.ln_1.weight", "gpt.gpt_inference.transformer.h.12.ln_1.bias", "gpt.gpt_inference.transformer.h.12.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.12.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.12.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.12.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.12.ln_2.weight", "gpt.gpt_inference.transformer.h.12.ln_2.bias", "gpt.gpt_inference.transformer.h.12.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.12.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.12.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.12.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.13.ln_1.weight", "gpt.gpt_inference.transformer.h.13.ln_1.bias", "gpt.gpt_inference.transformer.h.13.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.13.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.13.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.13.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.13.ln_2.weight", "gpt.gpt_inference.transformer.h.13.ln_2.bias", "gpt.gpt_inference.transformer.h.13.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.13.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.13.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.13.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.14.ln_1.weight", "gpt.gpt_inference.transformer.h.14.ln_1.bias", "gpt.gpt_inference.transformer.h.14.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.14.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.14.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.14.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.14.ln_2.weight", "gpt.gpt_inference.transformer.h.14.ln_2.bias", "gpt.gpt_inference.transformer.h.14.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.14.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.14.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.14.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.15.ln_1.weight", "gpt.gpt_inference.transformer.h.15.ln_1.bias", "gpt.gpt_inference.transformer.h.15.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.15.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.15.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.15.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.15.ln_2.weight", "gpt.gpt_inference.transformer.h.15.ln_2.bias", "gpt.gpt_inference.transformer.h.15.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.15.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.15.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.15.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.16.ln_1.weight", "gpt.gpt_inference.transformer.h.16.ln_1.bias", "gpt.gpt_inference.transformer.h.16.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.16.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.16.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.16.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.16.ln_2.weight", "gpt.gpt_inference.transformer.h.16.ln_2.bias", "gpt.gpt_inference.transformer.h.16.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.16.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.16.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.16.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.17.ln_1.weight", "gpt.gpt_inference.transformer.h.17.ln_1.bias", "gpt.gpt_inference.transformer.h.17.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.17.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.17.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.17.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.17.ln_2.weight", "gpt.gpt_inference.transformer.h.17.ln_2.bias", "gpt.gpt_inference.transformer.h.17.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.17.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.17.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.17.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.18.ln_1.weight", "gpt.gpt_inference.transformer.h.18.ln_1.bias", "gpt.gpt_inference.transformer.h.18.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.18.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.18.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.18.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.18.ln_2.weight", "gpt.gpt_inference.transformer.h.18.ln_2.bias", "gpt.gpt_inference.transformer.h.18.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.18.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.18.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.18.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.19.ln_1.weight", "gpt.gpt_inference.transformer.h.19.ln_1.bias", "gpt.gpt_inference.transformer.h.19.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.19.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.19.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.19.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.19.ln_2.weight", "gpt.gpt_inference.transformer.h.19.ln_2.bias", "gpt.gpt_inference.transformer.h.19.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.19.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.19.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.19.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.20.ln_1.weight", "gpt.gpt_inference.transformer.h.20.ln_1.bias", "gpt.gpt_inference.transformer.h.20.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.20.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.20.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.20.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.20.ln_2.weight", "gpt.gpt_inference.transformer.h.20.ln_2.bias", "gpt.gpt_inference.transformer.h.20.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.20.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.20.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.20.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.21.ln_1.weight", "gpt.gpt_inference.transformer.h.21.ln_1.bias", "gpt.gpt_inference.transformer.h.21.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.21.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.21.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.21.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.21.ln_2.weight", "gpt.gpt_inference.transformer.h.21.ln_2.bias", "gpt.gpt_inference.transformer.h.21.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.21.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.21.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.21.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.22.ln_1.weight", "gpt.gpt_inference.transformer.h.22.ln_1.bias", "gpt.gpt_inference.transformer.h.22.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.22.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.22.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.22.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.22.ln_2.weight", "gpt.gpt_inference.transformer.h.22.ln_2.bias", "gpt.gpt_inference.transformer.h.22.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.22.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.22.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.22.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.23.ln_1.weight", "gpt.gpt_inference.transformer.h.23.ln_1.bias", "gpt.gpt_inference.transformer.h.23.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.23.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.23.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.23.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.23.ln_2.weight", "gpt.gpt_inference.transformer.h.23.ln_2.bias", "gpt.gpt_inference.transformer.h.23.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.23.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.23.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.23.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.24.ln_1.weight", "gpt.gpt_inference.transformer.h.24.ln_1.bias", "gpt.gpt_inference.transformer.h.24.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.24.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.24.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.24.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.24.ln_2.weight", "gpt.gpt_inference.transformer.h.24.ln_2.bias", "gpt.gpt_inference.transformer.h.24.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.24.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.24.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.24.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.25.ln_1.weight", "gpt.gpt_inference.transformer.h.25.ln_1.bias", "gpt.gpt_inference.transformer.h.25.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.25.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.25.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.25.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.25.ln_2.weight", "gpt.gpt_inference.transformer.h.25.ln_2.bias", "gpt.gpt_inference.transformer.h.25.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.25.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.25.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.25.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.26.ln_1.weight", "gpt.gpt_inference.transformer.h.26.ln_1.bias", "gpt.gpt_inference.transformer.h.26.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.26.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.26.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.26.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.26.ln_2.weight", "gpt.gpt_inference.transformer.h.26.ln_2.bias", "gpt.gpt_inference.transformer.h.26.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.26.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.26.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.26.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.27.ln_1.weight", "gpt.gpt_inference.transformer.h.27.ln_1.bias", "gpt.gpt_inference.transformer.h.27.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.27.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.27.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.27.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.27.ln_2.weight", "gpt.gpt_inference.transformer.h.27.ln_2.bias", "gpt.gpt_inference.transformer.h.27.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.27.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.27.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.27.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.28.ln_1.weight", "gpt.gpt_inference.transformer.h.28.ln_1.bias", "gpt.gpt_inference.transformer.h.28.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.28.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.28.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.28.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.28.ln_2.weight", "gpt.gpt_inference.transformer.h.28.ln_2.bias", "gpt.gpt_inference.transformer.h.28.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.28.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.28.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.28.mlp.c_proj.bias", "gpt.gpt_inference.transformer.h.29.ln_1.weight", "gpt.gpt_inference.transformer.h.29.ln_1.bias", "gpt.gpt_inference.transformer.h.29.attn.c_attn.weight", "gpt.gpt_inference.transformer.h.29.attn.c_attn.bias", "gpt.gpt_inference.transformer.h.29.attn.c_proj.weight", "gpt.gpt_inference.transformer.h.29.attn.c_proj.bias", "gpt.gpt_inference.transformer.h.29.ln_2.weight", "gpt.gpt_inference.transformer.h.29.ln_2.bias", "gpt.gpt_inference.transformer.h.29.mlp.c_fc.weight", "gpt.gpt_inference.transformer.h.29.mlp.c_fc.bias", "gpt.gpt_inference.transformer.h.29.mlp.c_proj.weight", "gpt.gpt_inference.transformer.h.29.mlp.c_proj.bias", "gpt.gpt_inference.transformer.ln_f.weight", "gpt.gpt_inference.transformer.ln_f.bias", "gpt.gpt_inference.transformer.wte.weight", "gpt.gpt_inference.pos_embedding.emb.weight", "gpt.gpt_inference.embeddings.weight", "gpt.gpt_inference.final_norm.weight", "gpt.gpt_inference.final_norm.bias", "gpt.gpt_inference.lm_head.0.weight", "gpt.gpt_inference.lm_head.0.bias", "gpt.gpt_inference.lm_head.1.weight", "gpt.gpt_inference.lm_head.1.bias".
	Unexpected key(s) in state_dict: "gpt.conditioning_perceiver.latents", "gpt.conditioning_perceiver.layers.0.0.to_q.weight", "gpt.conditioning_perceiver.layers.0.0.to_kv.weight", "gpt.conditioning_perceiver.layers.0.0.to_out.weight", "gpt.conditioning_perceiver.layers.0.1.0.weight", "gpt.conditioning_perceiver.layers.0.1.0.bias", "gpt.conditioning_perceiver.layers.0.1.2.weight", "gpt.conditioning_perceiver.layers.0.1.2.bias", "gpt.conditioning_perceiver.layers.1.0.to_q.weight", "gpt.conditioning_perceiver.layers.1.0.to_kv.weight", "gpt.conditioning_perceiver.layers.1.0.to_out.weight", "gpt.conditioning_perceiver.layers.1.1.0.weight", "gpt.conditioning_perceiver.layers.1.1.0.bias", "gpt.conditioning_perceiver.layers.1.1.2.weight", "gpt.conditioning_perceiver.layers.1.1.2.bias", "gpt.conditioning_perceiver.norm.gamma".
	size mismatch for gpt.mel_embedding.weight: copying a param with shape torch.Size([1026, 1024]) from checkpoint, the shape in current model is torch.Size([8194, 1024]).
	size mismatch for gpt.mel_head.weight: copying a param with shape torch.Size([1026, 1024]) from checkpoint, the shape in current model is torch.Size([8194, 1024]).
	size mismatch for gpt.mel_head.bias: copying a param with shape torch.Size([1026]) from checkpoint, the shape in current model is torch.Size([8194]).

Environment

{
    "CUDA": {
        "GPU": ["NVIDIA A100-SXM4-80GB"],
        "available": true,
        "version": "11.8"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.1.0+cu118",
        "TTS": "0.20.1",
        "numpy": "1.22.0"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.9.18",
        "version": "#183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023"
    }
}

Additional context

No response

[atharvgarg]

I am encountering the same problem.

[KoljaB]

Same

[mhkarimi1383]

+1

[KoljaB]

Delete tts_models--multilingual--multi-dataset--xtts_v2 folder and let the model download again. Fixed the issue for me.

[jbang2004]

redownload the model can fix the problem,but i have found the wav's quality of this method is worse than the method of using api,don't know why

[min0ru]

I have checked how TTS api is loading the same exact model and it's different from code example from documentation: https://github.com/coqui-ai/TTS/blob/dev/docs/source/models/xtts.md

This is how i managed to load this model without errors:

from pathlib import Path

from TTS.tts.models import setup_model as setup_tts_model
from TTS.config import load_config


model_dir = Path("/home/user/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2")

config = load_config(model_dir / "config.json")
model = setup_tts_model(config)
model.load_checkpoint(config,
                      checkpoint_dir=model_dir,
                      eval=True,
                      # use_deepspeed=True,
                      )
model.to("cuda")

[erogol]

@Aya-AlJafari can you check the code above? It should have worked.

[Edresson]

redownload the model can fix the problem,but i have found the wav's quality of this method is worse than the method of using api,don't know why

This issue happens because the loaded model is not using the decoding parameters that are on config.json. You need to manually set them. Example:

    gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(audio_path=audio_path,gpt_cond_len=model.config.gpt_cond_len, max_ref_length=model.config.max_ref_len, sound_norm_refs=model.config.sound_norm_refs)

    out = model.inference(
        text=tts_text,
        language=lang,
        gpt_cond_latent=gpt_cond_latent,
        speaker_embedding=speaker_embedding,
        temperature=model.config.temperature, # Add custom parameters here
        length_penalty=model.config.length_penalty,
        repetition_penalty=model.config.repetition_penalty,
        top_k=model.config.top_k,
        top_p=model.config.top_p,
    )

[caffeinetoomuch]

I think the issue was caused by my mistake of not providing the right config file. After loading the config.json from fine-tuning checkpoint directory, the loading seems to be working now for both v1.1 and v2. Before closing the issue, I have the other questions regarding fine-tuning @Edresson

1. As I am trying to fine-tune XTTS with the single voice, how does trainer get voice reference for fine-tuning? Does speaker reference for testing sentence affect the fine-tuning results?
2. If XTTS was fine-tuned with the single voice dataset, does the inference still need the reference voice? of it there a way with the different fine-tuning config?

[Aya-AlJafari]

@erogol checked and it works.

[pseudotensor]

The above code fragment and my use started to fail with this. Was working fine until just last hour or so.

Unexpected key(s) in state_dict: "hifigan_decoder.waveform_decoder.ups.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.ups.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.ups.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.ups.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.ups.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.ups.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.ups.3.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.ups.3.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original1", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original0", "hifigan_decoder.waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original1". 
>>>