Engines

Engines

engine	model sources	generate	embed	transcribe	dimensions=
fake	fake	yes	yes	yes	yes
openai	openai	yes	yes	yes	yes
vllm	uc, path	yes	yes		yes
transformers	uc, path	yes	yes	yes
faster_whisper	uc, path			yes

Choosing an engine

engine="auto" (default):

model source	generate / embed	transcribe
OpenAI-compatible / Databricks	openai	openai
UC / path	vllm if CUDA and vLLM are available on the worker, else transformers	faster_whisper if installed, else transformers
fake	fake	fake

The choice happens on each worker: a CPU-only driver doesn't change what GPU executors use. Set engine= explicitly to pin one. Incompatible combinations (e.g. engine="vllm" with TranscribeJob) are rejected when the job is created.

engine_options are validated on the driver when the engine is known (explicit engine, or auto with a single candidate). Typos fail fast.

openai

AsyncOpenAI on a private event loop per worker process. It works inside notebooks that already run an event loop and keeps one connection pool for the whole run.

• max_concurrency requests in flight per worker, optional requests_per_minute.
• Retries with exponential backoff and jitter on 408/409/429/5xx, timeouts and connection errors (honouring Retry-After). Other 4xx errors fail the row at once.
• Generate: response_format=json_schema for output_schema. If the server rejects it, the engine switches to prompting for the rest of the run.
• Embed: batched requests, with per-row isolation when one input breaks a request.
• Transcribe: uploads the file (path or bytes); verbose_json for timestamps.

option	type	default	description
max_concurrency	int	16	Concurrent requests per worker process.
requests_per_minute	float | None	None	Request-start rate limit per worker process.
timeout_s	float	120.0	Per-request timeout in seconds.
max_retries	int	5	Retries on 408/409/429/5xx, timeouts and connection errors.
backoff_base_s	float	1.0	First retry delay; doubles per retry, with jitter.
backoff_max_s	float	60.0	Upper bound for a retry delay (also caps Retry-After).
structured_output	bool	True	Send output_schema as a json_schema response_format (falls back to prompting automatically if the server rejects it).
max_tokens_param	"max_tokens" | "max_completion_tokens"	"max_tokens"	Request field for Sampling.max_tokens (reasoning models need max_completion_tokens).
embed_max_inputs	int	256	Texts per embeddings request.
max_upload_mb	float	25.0	Audio files above this size are row errors.

vllm

Offline batched inference on CUDA GPUs: one vllm.LLM per task (generation, pooling).

• Structured output via StructuredOutputsParams (newer vLLM) or GuidedDecodingParams (older).
• Embeddings support dimensions.
• A call rejected because of one prompt (e.g. longer than max_model_len) is retried row by row, so only that row fails.
• All engine_options are passed to vllm.LLM(...). The common ones are type-checked.

option	type	default	description
tensor_parallel_size	int	1	GPUs per model replica (also sizes multiprocessing slots).
gpu_memory_utilization	float | None	None	Share of GPU memory vLLM may use.
max_model_len	int | None	None	Context length; longer prompts become row errors.
dtype	str | None	None	e.g. "bfloat16", "float16".
quantization	str | None	None	e.g. "awq", "gptq", "fp8".
trust_remote_code	bool	False	Allow custom model code from the model repository.

Other keys are passed through unchanged (not validated).

transformers

Hugging Face transformers on CPU or GPU (transformers 4.44+ and 5.x).

• Generate: causal LMs with the chat template.
• Embed: AutoModel + pooling read from sentence-transformers exports.
• Transcribe: the ASR pipeline.
• Micro-batches that run out of memory are split in half and retried.

option	type	default	description
device	str	"auto"	"auto" (cuda if available, else cpu), "cpu", "cuda:1", "mps".
device_map	str | None	None	e.g. "auto" to shard a big model over GPUs (needs accelerate).
dtype	"auto" | "float32" | "float16" | "bfloat16"	"auto"	auto = bfloat16/float16 on CUDA, float32 otherwise.
trust_remote_code	bool	False	Allow custom model code from the model repository.
micro_batch_size	int	8	Rows per forward pass (halved automatically on OOM).
pooling	"auto" | "mean" | "cls" | "max" | "last"	"auto"	Embedding pooling; auto = from the sentence-transformers export, else mean.
max_input_tokens	int | None	None	Embedding input truncation (default: model limit).
chunk_length_s	float	30.0	Speech recognition: chunk audio longer than this.
attn_implementation	str | None	None	e.g. "sdpa" or "flash_attention_2".

faster_whisper

CTranslate2 Whisper (distribatch[whisper]). CTranslate2 directories (model.bin) are used as they are. Hugging Face Whisper models are converted once per node into the model cache, which also needs distribatch[transformers].

option	type	default	description
device	str	"auto"	"auto", "cpu" or "cuda".
device_index	int | list[int]	0	GPU index, or several for one model on multiple GPUs.
compute_type	str	"default"	e.g. "float16", "int8_float16", "int8".
cpu_threads	int	0	CPU threads; 0 = CTranslate2 default.
num_workers	int	1	Files transcribed in parallel (threads) per process.
beam_size	int	5	Beam search width; 1 = greedy.
vad_filter	bool	False	Skip silence with Silero VAD.
batched	bool	False	VAD-based batched pipeline (faster long audio on GPU).
batch_size	int	8	Segments per batch with batched=True.
convert_quantization	str | None	"float16"	Weight type when converting HF Whisper to CTranslate2.

fake

No options. See Model Sources.