Can someone port the Google Colab training notebook over?

[rikochet9]

I'm very dependent on it for training voices and it's no longer working, apparently due to a Python 3.12 incompatibility.
rhasspy/piper#842

The new notebook mentioned in the thread doesn't look finished. The training step doesn't use of the variables set earlier in the notebook. Plus overall, it's not nearly as easy to configure as the old version was.

[KiON-GiON]

I had this one. I made it for people who needed it. I didn't post it in any discussion or issue because I was ashamed to say I got help from an LLM. But it works. Didn't fix the inference notebooks however. Colab's more limited lately for free users so I stopped using it. This one has automatic multi-speaker detection and optionally a way to provide direct URL to an external checkpoint. Just make sure the model quality and the base checkpoint quality match.

[rikochet9]

Thanks!

What settings have you gotten it working under? I've tried a few things with it and run into different problems.

Resuming checkpoint errors:

• _pickle.UnpicklingError: Weights only load failed. (1) In PyTorch 2.6, we changed the default value of the weights_only argument in torch.load from False to True
  • Did a def inside /content/piper1-gpl/src/piper/train/main.py to change torch.load to use weights_only False.
• Fixing that led to this error: error: Subcommand 'fit' does not accept option 'model.sample_bytes' Parsing of ckpt_path hyperparameters failed!
  • Apparently due to a mismatch in arguments between the version of Piper my checkpoint is from and the current version.

Trying to fine tune from hfc_male_medium:
NotImplementedError: Padding size 2 is not supported for 4D input tensor.

[KiON-GiON]

I only tried the Lessac high quality checkpoint so far because I'm used to finetune with it. I'll look into this issue.

edit: I think it's something happening with the download of the checkpoint. That error implies something bad with the file magic (i.e. the file begins with "<" instead of "PK".)
Maybe wget is met with an HTML page instead, which would explain the "<".
edit: Tried to download this specific checkpoint locally on my computer and I get:

--2025-12-05 02:46:48--  https://huggingface.co/datasets/rhasspy/piper-checkpoints/resolve/main/en/en_US/hfc_male/medium
/epoch%3D2785-step%3D2128064.ckpt
Resolving huggingface.co (huggingface.co)... 18.164.32.77, 18.164.32.106, 18.164.32.66, ...
Connecting to huggingface.co (huggingface.co)|18.164.32.77|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cas-bridge.xethub.hf.co/xet-bridge-[...]
--2025-12-05 02:46:48--  https://cas-bridge.xethub.hf.co/xet[...]
Connecting to cas-bridge.xethub.hf.co (cas-bridge.xethub.hf.co)|18.164.32.108|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

Yep, something strange happens, meanwhile I can download it fine on the browser.

[KiON-GiON]

Sorry for spamming. It happens with all checkpoints, or HF downloads in general. wget is broken with HF downloads right now so your best bet would be downloading the checkpoint locally on your computer with the browser, upload to your Google Drive and use the local drive path in the notebook.
Edit: Aria2 works so I'll change the downloader with that instead.
2nd edit: I fixed it, now it works properly again @rikochet9. The tensor thing is also fixed. I broke something because I touched what I shouldn't have.

[rikochet9]

It looks like I've run afoul of a behavior change.

• Previously, I ran with checkpoint_epochs = n, num_ckpt = -1, save_last = false. This made it save any number of checkpoint files.
• I see in the new logic in your notebook, it's no longer passing in a checkpoint_epochs parameter and instead maps it to every_n_epochs but only if num_ckpt > 0 or save_last = true.

So with no checkpoint_epochs value set and save_last = false, looks like my current run is not saving anything.

Me using -1 was itself due to saving multiple checkpoints being broken. Don't know if that might be fixed now or not: rhasspy/piper#749

[KiON-GiON]

Yep I really tried to do something with the new callback system but it was a pain to work with. I'll see what I can do.
Edit: I added a way to save all epochs if num_ckpt = -1, but depending on the number of checkpoint_epochs. If you want to save every epoch, set checkpoint_epochs to 1; or if you want to save every 50 epochs, set it to 50. works with validation enabled and disabled. num_ckpt >0 only saves the best checkpoints according to val_loss if validation is enabled, but the evaluation is triggered by every n epochs (checkpoint_epochs). It doesn't guarantee it will save every one of them. If the val_loss metric doesn't improve in the next n epochs, it will not save any checkpoint even if you train for a long time, or e.g. it will save only 3, even if you set num_ckpt to 20. I don't know if the issue you mentioned was a bug of the notebook, change of the default Lightning behavior when updating, or you were at the mercy of the validation metric, but you said you didn't had this issue in last year or so.

[rikochet9]

Thanks again! I’ll give it another go in the next day or two. Your explanation there would lineup with the behaviour I had back then. Training had likely plateaued. Was that documented somewhere?

[KiON-GiON]

Not in Piper at least, but maybe somewhere in Lightning's documentation. I also had something similar to what you described in the linked issue, but I just saved a "last.ckpt" since I did finetunes with small datasets.

[rikochet9]

Okay, completed some training and exported a model out.

I'm still not sure how that num_ckpt is supposed to work. I set checkpoint_epochs to 100 and num_ckpt to 3. It saved checkpoints for 100, 200, and 300 and then not 400 even though 300 doesn't sound fully trained yet.

This is what my val_loss looks like if it's any use. It doesn't cover the part that didn't save since, well, it didn't save...

[luiscarlos2000]

Is this the same notebook as this one https://github.com/ZachB100/Piper-Training-Guide-with-Screen-Reader or different than that? Guess I'll need to test it or maybe I don't know. I'm trying to port also that notebook to the new dependencies.

[rikochet9]

Yup, looks like the same notebook to me.

[luiscarlos2000]

Wow you responded so fast! I was going to do the same re-port the notebook from my friend Mateo whom I want to thank for, but maybe this new notebook will work. I was gonna continue my AI voice trayning but since big changes I've paused some, but hopefully this could wor. I've shared that new version with some friends, maybe they'll use it. KiON-GiON, you've somehow saved me. This community is thriving. Does this notebook have the latest languages added according to the Piper language list?

[rikochet9]

I think you need to tag @KiON-GiON for him to get a notification.

[KiON-GiON]

@luiscarlos2000 This update does the minimum to get the training notebook and exporter working again. I didn't mess with languages.

[luiscarlos2000]

Ah got it. Gonna test it.

[KiON-GiON]

Just to add something: I don't think you'll be able to continue old checkpoints since these have a lot of unused hyperparams/keys. Lightning will complain about this and won't load them. There's an untested theory, that if you want to continue them, you can use the option "finetune" with your old custom voices. Make sure to use the same settings (quality, batch size, etc.). This will transfer all the compatible parameters to your new checkpoint, but the epoch count, global steps and optimizers will reset, starting from epoch 0. If you use the same dataset and transcriptions, it will act as if you are continuing your voice.

[rikochet9]

Yes, I did successfully continue from a checkpoint using that method.