The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Here is a (tried-to-keep-short-but-long) update on this PR, summarizing some of the things discussed with @anton-l for public visibility.

1. Checkpoint loading: The model weight loads properly and produces roughly the same output as the fairseq model. On preliminary tests, I've found that torch.allclose passes with atol=1e-6. On this branch, the model can be loaded via

   from transformers import AutoModel
   model = AutoModel.from_pretrained("jaketae/fastspeech2-lj")
2. Grapheme-to-phoneme: Fairseq uses g2p_en. I've added this as a dependency via requires_backend in the tokenizer. One thing I'm not too sure about is whether to enforce this from the get-go at __init__, or only require it in the phonemize function.
3. Tokenizer: The tokenizer can be loaded as shown. For some reason, AutoTokenizer doesn't work at the moment, but this can likely be fixed by reviewing import statements. One consideration is that the fairseq code seems to add an EOS token at the end of every sequence, whereas HF's tokenizers do not by default (correct me if I'm wrong). In my code, I manually add the EOS token at the end for consistency.

   from transformers.models.fastspeech2 import FastSpeech2Tokenizer
   tokenizer = FastSpeech2Tokenizer.from_pretrained("jaketae/fastspeech2-lj")
4. Vocoder: (Note: This is not directly related to the source code in this PR.) Using Griffin-Lim for vocoding yields very poor audio quality and is likely a bad idea for demos. An alternative is to use neural vocoders like HiFi-GAN. I've added HiFi-GAN on the model hub, which can be imported via vocoder = AutoModel.from_pretrained("jaketae/hifigan-lj-v1", trust_remote_code=True). At the moment, for some reason the model weights do not load properly, debugging required.

I have some questions for discussion.

1. Model output: What should the output of the model be? At the moment, it returns a tuple of mel-spectrograms, mel-spectrogram after postnet, output lengths (number of spectrogram frames), duration predictor output, predicted pitch, and predicted energy. I assume we need to subclass BaseModelOutput for something like TTSModelOutput, and also optionally return loss when duration, pitch, and energy ground-truth labels are supplied in forward.
2. Mean and standard deviation: Fairseq's FastSpeech2 model technically does not predict a mel-spectrogram, but a normalized mean spectrogram according to mu and sigma values extracted from the training set. In fairseq's TTS pipeline, FS2 output is denormalized by some mu and sigma before being vocoded. I've added the mu and sigma values as register_buffer to the model, and created set_mean and set_std functions so that users can set these values according to their own training set. By default, mean and variance are set to 0 and 1 so that no rescaling is performed. Is this good design?
3. TTS pipeline: At the moment, there is only FastSpeech2Model, which outputs mel-spectrograms as indicated above. While people might want to use acoustic feature generators for mel-spectrogram synthesis, more often they will want to obtain raw audio in waveform. As mentioned, this requires a vocoder, and we need to discuss ways to integrate this. Some API ideas I have include:
   • Introducing a FastSpeech2ForAudioSynthesis class with an integrated vocoder
   • Adding a pipeline that supports full-fledged TTS via something like pipeline("text-to-speech", model="fastspeech2", vocoder="hifigan"). But this is again contingent on how we want to integrate the vocoder.

To-do's on my end (will be updated as I find more).

• Writing tests
• Documentation
• Cleaning up imports
• Debugging why AutoTokenizer fails
• Debugging why HiFi-GAN weights do not load

cc @anton-l @patrickvonplaten

@patrickvonplaten after a brief discussion with @jaketae, I think we should keep the spectrogram2waveform generators as separate models on the Hub for several reasons:

• the models for waveform generation are more or less interchangeable between different models (maybe with some finetuning) and are trained separately from the TTS models
• since the generators can be quite diverse, it would be cumbersome to support training for them, e.g. we probably don't have GANs or diffusion models in transformers for that very reason
• if I understand correctly, the generators usually don't support batched inference on inputs of varying lengths, so they're more suitable for pipelines rather than an integrated module

Having said that, I suggest that we finish this PR as a first step without adding any waveform generators. After that we can work on a TTS pipeline that runs inference on FastSpeech2 and decodes the output using either a Griffin-Lim synthesizer or any of the GAN models from the Hub (e.g. HiFiGAN) that we'll port.

Hi, may I ask a couple of questions:

1. Is there some estimated time of merging for this PR?
2. Are there any experiments/examples of training using this implementation Fastspeech2?
   And somewhat more general question:
3. Is there work in progress related to text-to-speech pipeline?

I'm working training/fine-tuning fastspeech2 models and would like to deploy using HF inference API.
Would be happy to help out with testing and contribute to pipeline and/or porting HiFiGAN or provide any assistance I can to speed things up with TTS

@anton-l, think it's not high priority but once you have a free day let's maybe add the model :-)

Wish I could have iterated faster to finish it earlier. Let me know if you need anything from me @anton-l @patrickvonplaten!

Wish I could have iterated faster to finish it earlier. Let me know if you need anything from me @anton-l @patrickvonplaten!

@jaketae if you want, would you like to finish this PR maybe ? E.g. fix the failing tests and add all the necessary code (including the vocoders) to the modeling file?

@anton-l I still think we should go ahead with this PR - it's been stale for a while now. Wdyt?

@patrickvonplaten definitely! I'm trying to find some time alongside other tasks to work on FS2 😅

Hey @anton-l @patrickvonplaten, hope you're doing well! I was wondering where FS2 fits within the transformers road map, and whether it would make sense for me to update this branch and wrap up the work. I feel responsible for leaving this stale, and I still think it would be cool to have this feature pushed out. Thank you!

Hey @jaketae,

We definitely still want to add the model - would you like to go ahead and update the branch?

I don't think anybody will have time to work on it soon. If anybody from the community wants to give it a shot, please feel free to do so!

I don't think anybody will have time to work on it soon. If anybody from the community wants to give it a shot, please feel free to do so!

Can I try working on this?

@JuheonChu I'm also interested in working on this, any interest in collaborating? Planning to get up to speed then open a fresh pr.

Hi @JuheonChu @connor-henderson, internally at HF we've been discussing what to do with this model. This PR has a lot of good work done already and FastSpeech 2 is a popular model, so it makes a lot of sense to add this to Transformers. If you want to help with this, that's great. 😃

There are a few different implementations of FastSpeech 2 out there and we'd like to add the best one to Transformers. As far as we can tell, there is a Conformer-based FastSpeech 2 model in ESPNet that outperforms regular FastSpeech 2, so that might be an interesting candidate. You can find the Conformer FastSpeech 2 checkpoints here.

In addition, PortaSpeech is very similar to FastSpeech 2 but it replaces the post-net and possibly a few other parts of the model. So that might be an interesting variation of FS2 to add as well.

We're still trying to figure out which of these models we'd prefer to have, but since you've shown an interest in working on this, I wanted to point out the direction we were thinking of taking these models internally. And I'd like to hear your opinions on this as well. 😄

P.S. Make sure to check out the implementation of SpeechT5. It has many features in common with FastSpeech2 (such as the vocoder and the generation loop) and so we'd want to implement FS2 mostly in the same manner.