Home

Parallel LLM Inference — GitHub Wiki

Project: parallel batch sentiment classification using PyTorch, Hugging Face Transformers, and spawned worker processes; also includes offline mode.

The default inference backend loads Qwen/Qwen2.5-0.5B-Instruct independently on each selected CUDA GPU (or on CPU). An offline --mock path replaces the LLM with a deterministic keyword heuristic so queueing and JSONL output can be exercised without model downloads.

Documentation

Start here	Architecture and internals	Operations
Getting Started	System Architecture (HLD)	CLI Reference
Data Contract	Low-Level Design (LLD)	GPU Deployment
Notebook Guide	Request Sequence	Testing and Validation
Source Code Map	Worker Lifecycle	Troubleshooting
Parallelism Explained	Multiprocessing and Queues	Security and Privacy
Model Loading and Generation	Dataset and Metrics	Publishing

Important scope

• Implemented: independent replica-per-GPU data-parallel batch inference, dynamic queue, text-length batching, JSONL output, optional synthetic labels and per-worker counts.
• Not implemented: DDP training, tensor/pipeline parallelism, model sharding, HTTP model serving, web UI, vector database, or production authentication.
• Mock-only evaluation: measures pipeline correctness, not model quality or true GPU throughput.

The uploaded DDP-training notebook is retained in legacy/ for completeness, but does not belong to this runtime architecture.