Crash and reboot on a 3090ti, yikes!

[chrisbward]

Ran the command;

python tortoise/do_tts.py --text "The vote has involved a series of individual workplace-based ballots across the UK and if nurses do not back action at a local level it is possible some hospitals and services will not be involved." --voice jlaw --preset fast

Power cut out and machine rebooted, scary!

[neonbjb]

Wouldn't be surprised if this was a power issue! 3090 and 3090Tis pull far more power than the spec sheet says. My 3090s spike at up to 550W.

[chrisbward]

I set the power cap to 300w and it still crashed!

sudo nvidia-smi -pl 300

What I've come to notice is that it only does it when I choose a voice other than "random", hope that helps!

[casonclagg]

Not sure if this is related, but the colab in this repo crashes before even downloading the models on A100 gpus. It was last working in colab for me Oct 31, 2022.

Update: Crashes on from tortoise.api import TextToSpeech, with A100 and T4

app.log: src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7f650a992d98

I'll try to run it locally this evening on a 3080 and see what happens...

[cason-soul]

Not sure if this is related, but the colab in this repo crashes before even downloading the models on A100 gpus. It was last working in colab for me Oct 31, 2022.

Update: Crashes on from tortoise.api import TextToSpeech, with A100 and T4

app.log: src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7f650a992d98

I'll try to run it locally this evening on a 3080 and see what happens...

The solution of !pip install transformers==4.19 right before installing requirements.txt in colab from issue #186 solves the issue of colab crashing.

[chrisbward]

Not sure if this is related, but the colab in this repo crashes before even downloading the models on A100 gpus. It was last working in colab for me Oct 31, 2022.

Update: Crashes on from tortoise.api import TextToSpeech, with A100 and T4

app.log: src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7f650a992d98

I'll try to run it locally this evening on a 3080 and see what happens...

Just wondering @casonclagg - where can I find the logs for this, just to confirm that's not my problem also? Thanks!

[sipvoip]

Wouldn't be surprised if this was a power issue! 3090 and 3090Tis pull far more power than the spec sheet says. My 3090s spike at up to 550W.

Yes, this works fine on my 3090ti

[chrisbward]

Wouldn't be surprised if this was a power issue! 3090 and 3090Tis pull far more power than the spec sheet says. My 3090s spike at up to 550W.

Yes, this works fine on my 3090ti

Windows? I'm trying this on Linux - Ubuntu 20.04

[chrisbward]

okay so, number of things I tried and managed to fix this issue;

• updated BIOS firmware on motherboard
• flicked the switch on the PSU from multi-rail to single

this may have been caused by the CPU hitting a temp, not really sure now, but no longer crashing