Getting Started

Getting Started

Prerequisites

• Python 3.11+ for the pinned Transformers 5.22.2 release (Python 3.12 recommended).
• For offline --mock: PyTorch and tqdm, with no Qwen download.
• For real inference: Transformers, network/cached weights, enough memory to hold a complete Qwen model per worker.
• For CUDA: a CUDA-compatible NVIDIA GPU driver and a CUDA-enabled PyTorch build.

Install on Windows PowerShell

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

Install a suitable CUDA PyTorch wheel from https://pytorch.org/get-started/locally/ if you intend to run GPU inference.

Run an offline smoke test

.\.venv\Scripts\python.exe parallel_llm_inference.py --mock --devices cpu cpu --input reviews.jsonl --output results_mock.jsonl --batch-size 4

Run real inference

.\.venv\Scripts\python.exe parallel_llm_inference.py --devices cuda:0 --input reviews.jsonl --output results_gpu.jsonl --batch-size 1 --max-input-tokens 192 --max-new-tokens 8

Verify

Open the output JSONL and confirm it has the same number and ordering of records as the input, plus llm_output and predicted_label. Real classification correctness is not guaranteed by the one-word prompt.