[Bug] latest-cuda image: CTranslate2 not compiled with CUDA support on ARM64 (aarch64) — NVIDIA DGX Spark / Grace Hopper

[Roknam]

Environment

Item	Details
Hardware	NVIDIA DGX Spark (Grace CPU + Hopper GPU)
Architecture	linux/arm64 (aarch64)
Docker image	ghcr.io/speaches-ai/speaches:latest-cuda
CUDA Version	12.6.3 (confirmed via nvidia-smi)
Container config	WHISPER__INFERENCE_DEVICE=cuda, WHISPER__COMPUTE_TYPE=float16

Problem Description

When running ghcr.io/speaches-ai/speaches:latest-cuda on an ARM64 machine (NVIDIA DGX Spark with Grace CPU + Hopper GPU), any transcription request results in a 500 Internal Server Error:

ValueError: This CTranslate2 package was not compiled with CUDA support

The server starts successfully, CUDA 12.6.3 is available on the host, and nvidia-smi works correctly. However, the ctranslate2 package inside the container is a CPU-only build — not compiled with CUDA support for arm64/aarch64.

Steps to Reproduce

docker run --rm -d \\
  -p 8000:8000 --gpus=all --name speaches \\
  -v hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \\
  -e WHISPER__INFERENCE_DEVICE=cuda \\
  -e WHISPER__COMPUTE_TYPE=float16 \\
  ghcr.io/speaches-ai/speaches:latest-cuda

curl http://localhost:8000/v1/audio/transcriptions \\
  -F "file=@audio.wav" \\
  -F "model=Systran/faster-whisper-large-v3" \\
  -F "language=ko"

Full Error Traceback

ValueError: This CTranslate2 package was not compiled with CUDA support
  File speaches/routers/stt.py line 183 in transcribe_file
  File speaches/executors/whisper/model_manager.py line 30 in _load_fn
  File faster_whisper/transcribe.py line 647 in __init__
    self.model = ctranslate2.models.Whisper(...)

Root Cause

The latest-cuda Docker image installs a CPU-only ctranslate2 wheel for arm64. No official ctranslate2[cuda] wheel exists for linux/arm64 on PyPI, so the build silently falls back to the CPU-only build.

Expected Behavior

ghcr.io/speaches-ai/speaches:latest-cuda should utilize GPU acceleration on ARM64 + CUDA platforms (NVIDIA DGX Spark, Grace Hopper Superchip).

Suggested Fix

For linux/arm64 builds, compile ctranslate2 from source with CUDA enabled using an NVIDIA NGC base image:

FROM nvcr.io/nvidia/cuda:12.6.0-cudnn-devel-ubuntu22.04
RUN pip install ctranslate2 --extra-index-url https://pypi.nvidia.com

Workaround

CPU mode (works but does not use the Hopper GPU):

docker run --rm -d -p 8000:8000 --name speaches \\
  -v hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \\
  -e WHISPER__INFERENCE_DEVICE=cpu \\
  ghcr.io/speaches-ai/speaches:latest-cuda

Additional Context

• Issue is specific to arm64. The latest-cuda image works correctly on amd64.
• Related NVIDIA Developer Forum: https://forums.developer.nvidia.com/t/speech-to-text-stt-api-docker-image-with-arm64-gpu-support/351324
• Speaches docs mention multi-platform build (linux/amd64,linux/arm64) but CUDA support per platform is not clarified.

[Roknam]

Additional Debug Information

Environment

• Hardware: NVIDIA DGX Spark (Grace Hopper Superchip, ARM64/aarch64)
• GPU: NVIDIA H100 NVL (Hopper architecture)
• OS: Ubuntu 22.04 (aarch64)
• Docker image: ghcr.io/speaches-ai/speaches:latest-cuda

Confirmed Behavior

The latest-cuda image runs but CTranslate2 falls back to CPU silently:

CTranslate2 CUDA not available, falling back to CPU

Verified via:

import ctranslate2
print(ctranslate2.get_supported_compute_types("cuda"))
# Returns empty or raises error on ARM64 build

Root Cause (Confirmed)

The CI/CD pipeline builds the linux/arm64 image without CUDA toolkit — likely because the build runner is a standard linux/amd64 host using QEMU emulation, where CUDA compilation for ARM64 is not straightforward.

The linux/amd64 variant compiles CTranslate2 with CUDA correctly because the build host natively supports it.

Suggested Build Fix

Use NVIDIA's native ARM64 CUDA base image:

# For arm64 CUDA support — use NGC base image
FROM nvcr.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 AS arm64-cuda-builder
# ... rest of build

Or leverage GitHub Actions' native ARM64 runners (now available) instead of QEMU emulation.

Current Workaround

docker run --rm -d -p 8000:8000 --name speaches \
  -v hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
  -e WHISPER__INFERENCE_DEVICE=cpu \
  ghcr.io/speaches-ai/speaches:latest-cuda

CPU mode works but does not utilize the Grace Hopper GPU — significant performance gap for production use on DGX Spark.

[Roknam]

I can confirm this issue on NVIDIA GB10 (Blackwell, sm_121) as well.

Additional Finding

For GB10 (Blackwell) users, there's an extra compatibility layer:

Container	Supported Compute Capability	GB10 Support
NGC 24.12	sm_80, sm_86, sm_90	❌
NGC 25.03+	sm_80, sm_86, sm_90, sm_121	✅

So even if CTranslate2 CUDA were fixed, NGC 24.x containers would still fail on Blackwell GPUs.

Workaround

Using nvcr.io/nvidia/pytorch:25.03-py3 with vanilla openai-whisper:

docker run --gpus all -d -p 8000:8000 \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --name whisper-gpu \
  -v hf-hub-cache:/root/.cache/huggingface \
  nvcr.io/nvidia/pytorch:25.03-py3 \
  sleep infinity

docker exec -it whisper-gpu bash
apt update && apt install -y ffmpeg
pip install openai-whisper fastapi uvicorn python-multipart

Full OpenAI-compatible server script: whisper_server.py

This successfully uses GPU acceleration on GB10:

CUDA: True, GPU: NVIDIA GB10
[large-v3] 34.9MB, duration=1720.0s, elapsed=180.2s  # ~9.5x realtime

[saikanov]

I can confirm this issue on NVIDIA GB10 (Blackwell, sm_121) as well.

Additional Finding

For GB10 (Blackwell) users, there's an extra compatibility layer:

Container Supported Compute Capability GB10 Support
NGC 24.12 sm_80, sm_86, sm_90 ❌
NGC 25.03+ sm_80, sm_86, sm_90, sm_121 ✅
So even if CTranslate2 CUDA were fixed, NGC 24.x containers would still fail on Blackwell GPUs.

Workaround

Using nvcr.io/nvidia/pytorch:25.03-py3 with vanilla openai-whisper:

docker run --gpus all -d -p 8000:8000
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864
--name whisper-gpu
-v hf-hub-cache:/root/.cache/huggingface
nvcr.io/nvidia/pytorch:25.03-py3
sleep infinity

docker exec -it whisper-gpu bash
apt update && apt install -y ffmpeg
pip install openai-whisper fastapi uvicorn python-multipart
Full OpenAI-compatible server script: whisper_server.py

This successfully uses GPU acceleration on GB10:

CUDA: True, GPU: NVIDIA GB10
[large-v3] 34.9MB, duration=1720.0s, elapsed=180.2s  # ~9.5x realtime

so this repo is not supporting DGX Spark yet huh, i can confirm this too, it took 40-100 second to transcribe 20 second audio, which mostlikely using cpu