Concepts

Concepts

Job (what to compute)  ──run(data, backend)──▶  Backend (where)  ──▶  Engine (how)  ──▶  Model (which)
GenerateJob                                      local                 openai               UC model
EmbedJob                                         multiprocessing       vllm                 directory / Volume
TranscribeJob                                    spark                 transformers         OpenAI-compatible API
                                                                       faster_whisper       fake

Jobs

A job is an immutable description of one task over rows: which model, which input columns, which parameters. There is one class per task:

• GenerateJob: chat/LLM generation from a {column} prompt template, optionally with structured output validated against a pydantic model.
• EmbedJob: one vector per row from a {column} text template.
• TranscribeJob: speech to text from a column with audio paths or bytes.

Jobs validate everything they can at construction (template syntax, model/engine compatibility, engine options), and run() validates the data (missing columns, output column clashes) before any work is scheduled. Configuration errors raise ConfigError. Problems with individual rows never raise.

Jobs are picklable. Schema classes defined in notebooks or in driver-only modules travel to workers automatically.

Model sources

Where the model comes from (Model Sources):

source	factory	URI
Unity Catalog registered model	models.uc(name, alias=/version=)	uc:cat.schema.model@alias
Directory / UC Volume / DBFS with HF weights	models.path(path)	/Volumes/..., dbfs:/..., ./weights
OpenAI-compatible API	models.openai(model, base_url=)	openai:<model>
Databricks Model Serving	models.databricks(endpoint)	databricks:<endpoint>
Fake (tests, dry runs)	models.fake(...)	fake

Anywhere a model is expected you can pass either a spec or its URI string.

Engines

The code that runs the model (Engines). engine="auto" (default) picks one per model source and task: openai for APIs; for local weights vllm on CUDA GPUs (generate, embed) else transformers; for transcription faster_whisper when installed, else transformers. Engine-specific settings go into engine_options={...} and are validated on the driver.

Backends

Where the rows are processed (Backends):

• local: in the calling process, sequentially. For tests, previews, small data.
• multiprocessing: a pool of worker processes on one machine, one per GPU by default.
• spark: Spark executors via mapInArrow. Works on classic clusters and on Spark Connect (shared/serverless compute).

Each worker process loads the model once and reuses it for all its batches.

Rows, batches and errors

Rows are grouped into batches of batch_size (default 32) for the engine. Whatever goes wrong with a row ends up in that row's <out>_error column:

• an unreadable input (e.g. corrupt audio);
• an API error after retries;
• invalid JSON for the schema;
• a crashed worker (multiprocessing).

The <out>_meta struct records tokens, audio duration, latency, attempts and the model. See Output Contract.