CGPT 104M

CGPT Release Notes

A complete one-epoch training run was performed for the 104m model on the approximately 10 GB FineWeb-Edu dataset. Training completed successfully after 58,874 optimizer steps.

Final losses:

• training loss: 3.302037;
• validation loss: 3.282484.

The model and checkpoints were saved in output/model-104m.

Experiment configuration

Parameter	Value
Model	104m
Input data	data/fineweb_edu_10gb.jsonl
Tokenizer	data/fineweb_edu_tokenizer_32k.json
Vocabulary size	32,000 tokens
Batch size	32
Sequence length	1,024 tokens
Epochs	1
Learning rate	3e-4
Minimum learning rate	3e-5
Warmup steps	0
Loss scale	1024
Validation fraction	10%
Validation interval	Every 5,000 steps
Validation batches	64

The training run was started with:

./build/cgpt_train \
  --input data/fineweb_edu_10gb.jsonl \
  --tokenizer data/fineweb_edu_tokenizer_32k.json \
  --output-dir output/model-104m \
  --model 104m \
  --vocab-size 32000 \
  --batch-size 32 \
  --sequence-length 1024 \
  --epochs 1 \
  --learning-rate 3e-4 \
  --min-learning-rate 3e-5 \
  --warmup-steps 0 \
  --loss-scale 1024 \
  --validation-fraction 0.1 \
  --validation-interval 5000 \
  --validation-batches 64 \
  --save-avg-loss

Using the CLI

The cgpt_cli executable supports both one-shot generation and an interactive REPL.

Start interactive mode

Run the CLI without --prompt:

./build/cgpt_cli \
  --model outputdg/step-58874 \
  --device cuda

Enter a prompt after the You > prompt and press Enter:

You > The future of local AI is

The generated text will be displayed below. You can then enter another prompt without restarting the application.

REPL commands

Command	Description
/help	Display available commands
/params	Show current generation parameters
/set <parameter> <value>	Change a parameter during the session
/exit	Exit the CLI
/quit	Exit the CLI

Examples:

/set temperature 0.7
/set top-k 40
/set top-p 0.95
/set max-new-tokens 256
/set repetition-penalty 1.1
/params

One-shot generation

Use --prompt to generate text once and exit:

./build/cgpt_cli \
  --model outputdg/step-58874 \
  --prompt "The future of local AI is" \
  --max-new-tokens 128 \
  --temperature 0.8 \
  --top-k 40 \
  --top-p 0.95 \
  --seed 42 \
  --device cuda

The most useful generation options are:

• --max-new-tokens: maximum number of generated tokens;
• --temperature: controls randomness;
• --top-k: limits sampling to the top k tokens;
• --top-p: enables nucleus sampling;
• --repetition-penalty: reduces repeated text;
• --no-repeat-ngram: blocks repeated n-grams;
• --seed: makes generation reproducible;
• --device auto|cpu|cuda: selects the execution device.

To display all available options:

./build/cgpt_cli --help

Data preparation

The BPE tokenizer was trained on 1,024 MiB of data. This took 227.26 s at an average speed of 139.66 iterations/s.

The complete input file was then tokenized:

• processed data size: 9,843,688,080 bytes;
• tokenization time: 22.62 s;
• throughput: 414.98 MiB/s;
• generated tokens: 2,143,562,534.

Training progress

Step	Training loss	Validation loss
5,000	3.910492	3.908841
10,000	3.702029	3.707374
15,000	3.562318	3.607797
20,000	3.399266	3.540929
25,000	3.500671	3.487137
30,000	3.451362	3.441219
35,000	3.431984	3.398644
40,000	3.370052	3.362667
45,000	3.197763	3.331056
50,000	3.220727	3.306455
55,000	3.411847	3.289633
58,874	3.302037	3.282484

Validation loss chart

xychart-beta
    title "Validation Loss"
    x-axis "Optimizer step" [5000, 10000, 15000, 20000, 25000, 30000, 35000, 40000, 45000, 50000, 55000, 58874]
    y-axis "Loss" 3.2 --> 4.0
    line [3.908841, 3.707374, 3.607797, 3.540929, 3.487137, 3.441219, 3.398644, 3.362667, 3.331056, 3.306455, 3.289633, 3.282484]

Validation checkpoints were saved at steps 5000, 10000, 15000, 20000, 25000, 30000, 35000, 40000, 45000, 50000, 55000, and 58874.

The actual training time was approximately 12,741.46 s, or 3 hours 32 minutes, with a stable throughput of about 4.62 steps/s.

If CLI crash

Please send me your CPU model, GPU model if you have one, RAM amount. On my device evything works fine so its hard do fix bugs without any device data