Backends

Backends

All backends run the same per-row pipeline:

1. build the request (template, audio input);
2. call the engine in batches;
3. post-process the result (schema validation, vector checks);
4. map any error to the row.

They differ in where that happens.

local

Sequential, in the calling process. Reference behaviour, used by preview() and tests. API engines still send max_concurrency requests concurrently within a batch. The engine works inside notebooks that already run an asyncio event loop.

multiprocessing

A pool of worker processes on one machine (spawn start method, safe with CUDA).

• Model loading: each worker loads the model once at start-up. A failure to load (e.g. missing credentials) aborts the run with that error.
• GPU placement: GPUs are detected from CUDA_VISIBLE_DEVICES or nvidia-smi (torch is not imported on the driver) and split into slots of tensor_parallel_size GPUs. Each worker gets one slot through its own CUDA_VISIBLE_DEVICES, set before CUDA initializes. By default there is one worker per slot, and an explicit workers= shares slots round-robin. Without GPUs local models get 1 worker. API models get min(4, CPUs) workers, each with max_concurrency requests in flight.
• Worker crashes: if a worker dies (out-of-memory kill, segfault in a decoder), only the chunks it was processing are affected. They are re-run one at a time, and a chunk that kills a worker on its own is split in half until the offending row is found. That row gets WorkerCrashedError, and all other rows complete. Pool restarts are limited to 10 per run; beyond that the remaining rows are marked failed.
• Progress: logged on the distribatch logger every 10 s, plus a summary at the end.
• Scripts: plain Python scripts (not notebooks) must guard the call with if __name__ == "__main__":, as with any spawn-based pool.

spark

DataFrame.mapInArrow. It works on classic clusters and on Spark Connect (Databricks shared and serverless compute) and needs no SparkContext or RDD API.

• Columns: only the columns the job reads are converted to Python. All other columns pass through as the original Arrow data, keeping exact types (binary, structs, arrays, timestamps).
• Output schema: computed on the driver before the job starts. See Output Contract.
• Model loading: each Spark Python worker loads the model once, as Python workers are reused across tasks.
• Parallelism = concurrently running tasks × the engine's own concurrency. Use partitions= to set the number of tasks. For API engines size it so that partitions × max_concurrency stays within the endpoint's rate limits. For GPU engines use one task per GPU (see Databricks Guide).
• Batch size: Arrow batches follow spark.sql.execution.arrow.maxRecordsPerBatch (default 10 000 rows), and batch_size splits them further for the engine.
• Failures: row problems are row errors. A model that can't load fails the Spark task, and so the job after Spark's task retries. A dead Python worker is retried by Spark. There is no poison-row bisection as in multiprocessing.
• Lazy results unless you use a checkpoint. See Running Jobs.
• Requirements: distribatch must be installed on the cluster (notebook %pip installs reach the executors). Schema classes from notebooks or repo files are shipped automatically.