Problem Statement
Support for onnxruntime-gpu exists but is exclusively for NVIDIA. The vast majority use AMD but there is no rocm support through use of onnxruntime-rocm. My CPU isn't bad, it's a 5800x but even doing a modest sized ePub file or text it takes hours. But if a GPU could be used instead then it can run real-time and potentially make even 500k word ePubs or longer take less than an hour.
Proposed Solution
Alternative Solutions
Making an NCNN conversion of the model could work, I've seen it done on other AI models so it should be possible. NCNN has worked on my AMD GPU before so it could work as (AMD/intel) opensource-driver version.
User Impact
GPU acceleration on AMD GPU's which is 90% of the Linux user base.
Implementation Ideas
Theoretically it should be exactly the same as the process for NVIDIA (onnxruntime-gpu) but you use onnxruntime-rocm instead. You can even set the py project dependency line to have "kokoro-onnx[gpu]==0.3.9" be "kokoro-onnx[rocm]==0.3.9" instead.
You could even have the [gpu] part just detect if you have cuda or rocm installed and use that as a possible reference? But I am not sure.
Problem Statement
Support for
onnxruntime-gpuexists but is exclusively for NVIDIA. The vast majority use AMD but there is no rocm support through use ofonnxruntime-rocm. My CPU isn't bad, it's a 5800x but even doing a modest sized ePub file or text it takes hours. But if a GPU could be used instead then it can run real-time and potentially make even 500k word ePubs or longer take less than an hour.Proposed Solution
Alternative Solutions
Making an NCNN conversion of the model could work, I've seen it done on other AI models so it should be possible. NCNN has worked on my AMD GPU before so it could work as (AMD/intel) opensource-driver version.
User Impact
GPU acceleration on AMD GPU's which is 90% of the Linux user base.
Implementation Ideas
Theoretically it should be exactly the same as the process for NVIDIA (
onnxruntime-gpu) but you useonnxruntime-rocminstead. You can even set the py project dependency line to have"kokoro-onnx[gpu]==0.3.9"be"kokoro-onnx[rocm]==0.3.9"instead.You could even have the [gpu] part just detect if you have cuda or rocm installed and use that as a possible reference? But I am not sure.