CPU and GPU Acceleration

CPU and GPU Acceleration

A2M can execute the same scorer and attribute models through ONNX Runtime on CPU, CUDA, or DirectML. Provider choice affects where compatible model operations run and how sections are batched. It does not select a different transcription engine.

Comparison

Mode	Extra managed runtime	Section batch	Intended use
CPU	No optional GPU pack	1	Default and most compatible path
CUDA	CUDA runtime pack; compatible NVIDIA CUDA/cuDNN components may also be required	Configurable	NVIDIA GPU acceleration
DirectML	DirectML runtime pack	Configurable	Windows GPU acceleration through DirectML

This doesn't promise that GPU mode is faster on every system. Session startup, recording length, GPU generation, drivers, transfers, batch size, CPU performance, and model/provider compatibility all affect the result.

Runtime packs

CUDA and DirectML use separate managed runtime packs under the user's A2M application-data directory. A pack contains a provider-specific ONNX Runtime build and metadata identifying:

• provider;
• ONNX Runtime version;
• Python ABI;
• required package layout and provider artifacts.

A2M checks archive size and SHA-256, extracts within a bounded staging area, validates metadata and files, and runs a helper-process provider test before making the pack active.

Why validation uses a helper process

ONNX Runtime is a native backend. Once a runtime has been imported into a process, safely replacing it with another provider build is not always possible. A2M therefore performs inventory and activation checks in a separate process and can request an application restart when the selected runtime path changes.

Validation checks more than whether a DLL exists. Depending on the stage, it can:

• import the selected ONNX Runtime package;
• verify the requested execution provider is listed;
• create scorer and attribute sessions with the real model;
• run small inference probes;
• confirm that the session's active provider is the requested GPU provider.

CPU resource setting

The Low, Balanced, High, and Max CPU settings select approximately 25%, 50%, 75%, and 100% of detected logical processors, rounded upward with at least one worker. The active worker count is supplied to ONNX Runtime thread options and used by numerical thread controls.

These percentages describe worker allocation, not a guaranteed percentage in Windows Task Manager.

GPU memory setting

GPU memory choices map to section batch sizes bounded by a hardware-derived maximum. The maximum uses detected dedicated GPU memory when available. It is a coarse resource control, not an exact VRAM reservation.

If a scorer batch fails, A2M halves/reduces the effective batch and retries. The smaller working value is cached for the active bundle/provider session.

Fallback behavior

There are several distinct cases:

1. No matching pack: A2M can offer to download the selected provider pack. The attempted conversion does not continue while that setup occurs.
2. Runtime change needs restart: A2M asks for a restart before conversion can use the new backend.
3. Validation still running: The conversion is paused and the user may need to try again after validation finishes.
4. Provider cannot activate: A2M can change the preference back to CPU and report the reason.
5. Recognized GPU failure during conversion: A2M changes to CPU mode for subsequent work, but it does not automatically rerun the failed file.

That last distinction matters: “automatic CPU fallback” means the application recovers its selected mode; it does not always mean the current conversion silently restarts from the beginning.

Accuracy and parity

CPU, CUDA, and DirectML are intended to run the same model and decoding logic. Small numerical differences between execution providers are possible. Exact parity should be checked on the relevant runtime versions and hardware when it matters.

See CUDA Setup and Troubleshooting and DirectML Setup and Troubleshooting.