Dataset and Metrics

Datasets and Observability

The included reviews.jsonl contains 48 synthetic product reviews with sentiment labels. This is a functional example, not an independently curated quality evaluation set.

The parent reports worker initialization time, inference processing time, throughput in rows per second, an approximate output-token rate, and number of rows processed per worker. When rows include label equal to positive or negative, it reports strict normalized accuracy.

Interpretation: mock accuracy is an attribute of a simple keyword heuristic on synthetic data. Model-load time is separate from inference time. Meaningful GPU benchmarks should use the same dataset, model, batch sizing, generation caps and warmup procedure while comparing single GPU vs multiple GPUs.

flowchart TD
    S["Start / worker spawn"] --> L["Measure all-worker model load time"]
    L --> E["Release go_event"]
    E --> R["Measure inference elapsed time"]
    R --> M["Records per worker + total rows + output tokens"]
    M --> T["rows/sec + approx tokens/sec"]
    T --> A["Optional accuracy for labeled JSONL"]
    A --> N["Interpret mock and real model benchmarks separately"]

Loading