Output Contract

Output Contract

run() returns the input with three columns added (here output_col="output"):

column	content
output	the result, or null when the row failed
output_error	null on success, otherwise "<ErrorClass>: <message>"
output_meta	metrics, same fields for every task (below)

Rows keep their order (pandas: also the index; Spark: all input columns and types). Rows are never dropped: every input row has an output row.

Value types

job	local / multiprocessing	Spark
GenerateJob	str	string
GenerateJob(output_schema=Model)	dict (JSON-native)	struct<...> mapped from the schema
EmbedJob	list[float]	array<float>
TranscribeJob	str	string
TranscribeJob(timestamps=True)	{"text", "language", "segments": [{"start", "end", "text"}]}	struct<text:string, language:string, segments:array<struct<start:double, end:double, text:string>>>

output_meta

field	Spark type	meaning
prompt_tokens	bigint	input tokens (generate, embed); null when unknown
completion_tokens	bigint	generated tokens (generate)
audio_seconds	double	duration of the audio (transcribe)
latency_ms	double	time spent on the row (batch time divided by rows for local engines)
attempts	bigint	API attempts incl. retries; 0 = not sent (input error, or reused by dedupe)
model_id	string	model that produced the row
finish_reason	string	"stop", "length", ... (generate)

Errors

Configuration problems raise before any work: ConfigError (bad template, unknown column, incompatible engine, invalid engine_options, checkpoint of another job, …) and MissingDependencyError (an extra is not installed). A model that can't be loaded (missing weights, missing credentials on a worker) also raises and aborts the run.

Everything else is a row error in <out>_error:

error class	when
PromptRenderError	the template could not be filled for this row (missing key, bad format spec)
InvalidInputError	unusable input: empty text to embed; null/empty/unreadable/undecodable audio; audio over the upload limit
SchemaValidationError	the model's answer is not valid JSON or does not match output_schema
OutputTypeError	Spark only: a value does not fit the column type (possible with a dict schema)
EngineError	the engine failed for this row or batch (batch failed: ...), returned no or invalid output
WorkerCrashedError	multiprocessing: the worker process died on this row, or the pool restart limit was reached
API / framework classes	e.g. RateLimitError after all retries, BadRequestError, ValueError from vLLM

All library exceptions derive from DistribatchError. dib.summarize(out).errors counts row errors by class (Statistics and Costs).

Structured output on Spark

The struct type is derived from the JSON schema of output_schema on the driver, before the job starts:

JSON schema (pydantic)	Spark type
str, Literal[...], enums, dates, datetimes	string (dates as ISO strings)
int	bigint
float	double
bool	boolean
list[X]	array<X>
dict[str, X]	map<string, X>
nested model	struct<...>
X | None	X (all fields are nullable)
unions of different types, free-form objects (dict, Any), recursion deeper than 32 levels	string containing JSON

Values are coerced to the declared type on the executors. A value that doesn't fit (e.g. text where the schema says integer, only possible with a dict schema) makes that row fail with OutputTypeError instead of failing the Spark job.