WhisperVC Inference

Inference-only runtime for Chinese and English WhisperVC models. See the paper.

Install

Use Python 3.10 or newer. Install matching PyTorch and TorchAudio builds for your CUDA environment, then run:

pip install -e .

Download models

The model repository uses automatic gated access. Request access in the browser, authenticate locally, and download a language bundle:

hf auth login
python scripts/download_models.py \
  --repo-id RaydonLiu/WhisperVC \
  --language cn \
  --accept-wespeaker-license

Run the command again with --language en for English. Each bundle contains Stage2, HiFiGAN, Gate, and Whisper Flamingo encoder/adapter weights.

Verify a downloaded bundle with python scripts/verify_checkpoint.py --language cn (or en).

The WeSpeaker checkpoint is downloaded separately from the official release and is not redistributed. Review its upstream model and VoxBlink2 dataset license before accepting the terms.

Run one audio file

python scripts/infer_single.py \
  --input path/to/whisper.wav \
  --output path/to/generated.wav \
  --mode w2n \
  --ref-audio path/to/reference_speaker.wav \
  --language cn

w2n treats the input as whisper speech, vc treats it as normal speech, and auto uses the bundled Gate. If --ref-audio is omitted, the input is also used as the speaker reference.

Citation

If you find WhisperVC useful in your research, please cite:

@article{liu2025whispervc,
  title = {WhisperVC: Decoupled Cross-Domain Alignment and Speech Generation for Low-Resource Whisper-to-Normal Conversion},
  author = {Dong Liu and Juan Liu and Wei Ju and Yao Tian and Ming Li},
  journal = {arXiv preprint arXiv:2511.01056},
  year = {2025}
}

License

The source code is licensed under Apache-2.0. WhisperVC model weights are licensed under CC BY-NC-SA 4.0 for non-commercial use. Training data: AISHELL6-Whisper, wTIMIT, and LibriTTS.