Notebook Guide

Inference Notebook Guide

Open Parallel_LLM_Inference_Colab_Kaggle.ipynb in Colab, Kaggle, JupyterLab or VS Code.

• A notebook cell uses %%writefile parallel_llm_inference.py to create the real executable CLI.
• Offline cells generate a synthetic reviews.jsonl, run python parallel_llm_inference.py --mock ... as a subprocess and compare row order.
• Real Qwen execution requires changing RUN_REAL_INFERENCE = False to True and installing the model dependencies.
• A separate cell optionally compares one- versus two-GPU throughput (requires two visible GPUs).

Launching spawned worker processes from a standalone Python file avoids common Jupyter/Windows pickling problems. Notebook execution directly via %run is not recommended for the multiprocessing worker loop.

Do not confuse with uploaded legacy/parallel_llm_inference_colab.ipynb, which teaches distributed training using DDP, not inference.