Pre-Training data

[wang99711123]

Hello, thanks for your wonderful work! I am trying to reproduce your work. Which slice of OLMo 2 did you use for pretraining? Or a randomly selected 330B~ data?

[alisawuffles]

We tokenized the entire Olmo2 pretraining corpus, and in the training config, we set the number of training steps, sequence length, and global batch size, which determine the final number of tokens used. E.g., for UW/OLMo2-8B-SuperBPE-t180k, the settings were

max_duration: 107972
max_sequence_length: 3000
global_train_batch_size: 1024

(Note that for some reason, the Olmo2 script trains for 10 more steps than specified in the config.)

Let me know if you have any further questions!