GPU Deployment

One Replica Per GPU

Each selected GPU owns its own full model copy, tokenizer state, runtime buffers and KV cache. GPUs never combine their memory to load a model. For a single 4 GB GPU, begin with --batch-size 1, --max-input-tokens 192, --max-new-tokens 8. A second GPU can improve throughput for sufficiently large batches, but total speedup depends on I/O, model loading, memory and GPU capabilities.

Examples:

python parallel_llm_inference.py --devices cuda:0 --input reviews.jsonl --batch-size 1 --output one_gpu.jsonl
python parallel_llm_inference.py --devices cuda:0 cuda:1 --input reviews.jsonl --batch-size 1 --output two_gpu.jsonl

Never duplicate cuda:0 in --devices: the executable rejects duplicate CUDA assignments. --devices cpu cpu --mock is supported for offline testing.

flowchart TB
    P["Host parent + input batch queue"] --> G0["Spawn worker 0"]
    P --> G1["Spawn worker 1"]
    subgraph GPU0["GPU 0 - independent memory"]
      G0 --> M0["Full Qwen model replica"]
      M0 --> B0["Tokenize / generate batch"]
    end
    subgraph GPU1["GPU 1 - independent memory"]
      G1 --> M1["Full Qwen model replica"]
      M1 --> B1["Tokenize / generate batch"]
    end
    B0 --> R["Parent result collector"]
    B1 --> R
    R --> O["Ordered JSONL output"]

Loading