Request Sequence

Low-Level Runtime Sequence

The worker reports ready only after model initialization (or immediate mock setup). The parent starts the producer thread, raises the shared go event, and receives batch responses. Responses may arrive out of order; final file writes use original record indices. Errors are sent to the result queue and cause parent-side termination/cleanup.

sequenceDiagram
    autonumber
    actor User
    participant Parent as Parent / main()
    participant Task as Task Queue
    participant W0 as Worker 0
    participant W1 as Worker 1
    participant Result as Result Queue
    participant FS as Output JSONL
    User->>Parent: --input reviews.jsonl --devices cuda:0 cuda:1
    Parent->>Parent: read_jsonl, validate, sort, form batches
    Parent->>W0: spawn + load model replica
    Parent->>W1: spawn + load model replica
    W0-->>Result: ready
    W1-->>Result: ready
    Parent->>Parent: await all ready, set go_event
    Parent->>Task: enqueue batches + one sentinel per worker
    par parallel processing
      W0->>Task: get batch
      W0->>W0: generate_batch or mock_generate
      W0-->>Result: batch(ids, outputs, token_count)
    and
      W1->>Task: get batch
      W1->>W1: generate_batch or mock_generate
      W1-->>Result: batch(ids, outputs, token_count)
    end
    Parent->>Result: receive until every row accounted for
    Parent->>Parent: restore original input order
    Parent->>FS: write llm_output + predicted_label
    Parent-->>User: throughput / optional accuracy

Loading