Data Pipeline

Data Pipeline and Sharding

Input contract

The built-in list contains 16 educational example sentences. --corpus-file accepts UTF-8 text with one training example per non-blank line. Samples are repeated using --repeat (default 4). Data is kept in memory as a Python list.

flowchart TB
    Text["Text lines x repeat"] --> DS["IndexedTextDataset: (record_id, text)"]
    DS --> Split{"World size > 1?"}
    Split -->|yes| Sampler["DistributedSampler, shuffle seed 2026; set_epoch"]
    Split -->|no| Shuffle["Single-process DataLoader shuffle"]
    Sampler --> R0["Rank 0 record indices"]
    Sampler --> R1["Rank 1 record indices"]
    R0 --> Loader0["DataLoader, batch_size per rank"]
    R1 --> Loader1["DataLoader, batch_size per rank"]
    Shuffle --> LoaderSingle["Single DataLoader"]
    Loader0 --> Collator["Tiny byte collator or Qwen tokenizer collator"]
    Loader1 --> Collator
    LoaderSingle --> Collator
    Collator --> Device["Tensor batches on CPU or local CUDA GPU"]

Loading

Tiny byte-level collator

1. Encode sample text as UTF-8 bytes; retain at most seq_len - 1 bytes.
2. Map byte values 0..255 to token IDs 2..257 and append EOS token 1.
3. Pad with ID 0 to seq_len.
4. Clone input_ids to labels, replacing PAD targets with -100 (ignored in cross-entropy).

Qwen tokenizer collator

The pretrained tokenizer performs truncation and right padding to seq_len, and supplies an attention_mask. Labels are copied from token IDs with padded positions masked to -100.

Limitations: No validation split, document deduplication, dataset streaming, dynamic sequence bucketing, assistant-only loss mask, or real task-quality evaluation is implemented. Using a realistic training dataset requires separate design work.