Troubleshooting

Troubleshooting

MissingDependencyError: 'vllm' is required … pip install 'distribatch[vllm]'

The engine or feature needs an extra. Install it (on Databricks with %pip in the notebook, or as a cluster library).

ConfigError: job references column(s) missing from the input: …

A {field} in the template, or audio_col, is not a column of the data. Field names are case-sensitive.

ConfigError: output column(s) already exist in the input

Choose another output_col, or drop the old output columns first (e.g. re-running on already scored data).

ConfigError: environment variable X is not set on this worker

Executors don't see notebook secrets or the notebook's authentication. Set the variable in the cluster's environment variables (secret references like {{secrets/scope/key}} work). See Databricks Guide.

ConfigError: cannot resolve Unity Catalog model …

The worker can't reach the model registry. Use models.uc(..., stage_dir="/Volumes/..."), or provide DATABRICKS_HOST/DATABRICKS_TOKEN to executors.

ConfigError: checkpoint … belongs to a different job

The job changed (prompt, schema, model, sampling…) since the checkpoint was written. Use a new checkpoint directory, or delete the old one to start over.

dimensions= is not supported by engine transformers

Use an engine that supports shorter embeddings (engine="vllm", or an API model), or drop dimensions.

Many SchemaValidationError rows

• Inspect the raw answers: the error contains the start of the model output.
• Make the prompt explicit about the fields, simplify the schema (fewer nested optional fields), and use a model that supports structured output (API with json_schema, or vLLM).
• Raise Sampling(max_tokens=...): truncated JSON is a common cause, visible as finish_reason="length" in the meta.

Many RateLimitError rows

Lower partitions or max_concurrency, set requests_per_minute, or raise max_retries/backoff_max_s. With a checkpoint, simply re-run later: only failed rows are sent again.

RuntimeError: An attempt has been made to start a new process…

Plain scripts using backend="multiprocessing" need an if __name__ == "__main__": guard around the code that runs the job.

WorkerCrashedError: worker process died on this row

That row made a worker process die, usually a native crash in a decoder or library, or out-of-memory. All other rows were processed. Check the input (e.g. a corrupt audio file). For OOM, reduce batch_size/micro_batch_size or the number of workers per GPU.

The Spark job ran inference twice

Without a checkpoint, the result is lazy and every action recomputes it. Write the result once or use a checkpoint. See Running Jobs.

All models land on GPU 0 / CUDA out of memory with several tasks per node

Set spark.task.resource.gpu.amount so Spark assigns GPUs to tasks. distribatch pins each Python worker to its task's GPUs. See Databricks Guide.

Audio: cannot decode audio … (install ffmpeg for m4a/aac/webm)

libsndfile reads wav/flac/ogg/mp3/aiff. Install ffmpeg on the workers for other formats, or convert the files.

Seeing what happens

import logging
logging.basicConfig()
logging.getLogger("distribatch").setLevel(logging.INFO)

This logs progress (multiprocessing), GPU pinning, checkpoint waves, pool restarts and the end-of-run summary.