compare fastText versus BERT for quality classifiers

[percyliang]

fastText is often used to train a quality classifier to select data for pre-training (e.g., in OLMo, DCLM, etc.) because it's, well, fast, and easy to use. There are times when the predictions of the fastText classifier doesn't really make sense. We have much stronger models like BERT now - could those be better? We want to do an apples-to-apples comparison where we're using the same positive/negative examples, extracting the same number of tokens, and fine-tuning the same model.

Now of course one can use even stronger models like Llama 70B, but usually those get distilled into something that's faster to run.

[dlwh]

• train bert
• annotate
• don't train lm for now

[RohithKuditipudi]

Plan: ablate #614 , #274 , and #605 swapping BERT for fasttext.

[RohithKuditipudi]

Artifacts:

1. Wandb report.
2. Experiment browser.

Summary:
We ablated #274 with a BERT classifier. In particular, we trained both a BERT and fastText classifier using 100k MMLU examples as the positive set and 100k documents from the DCLM pool as the low quality set. We then filtered at varying thresholds (keeping top x% for x = {1,5,10,20}).

We mainly used x = {1,5} to run a (very) small internal study where a few members of the Marin team blindly rated the different filtered datasets to see if there was a qualitative difference. In general, the team preferred BERT to fastText, but also preferred both BERT and fastText top 1% to e.g. BERT top 20%. Interestingly, prior work found that human evaluations did not correlate well with downstream performance (Appendix J). In particular, they ask humans to rate 500 documents and then measure agreement between the human ratings and various fastText quality classifiers; they find that the degree to which a fastText classifier aligns with human ratings is not predictive of how well a model trained on data filtered by the classifier will perform. Though not apples to apples, at least in this specific case our (very) small study suggests there is a qualitative (i.e., human-perceptible) difference between in the quality of BERT vs. fastText-filtered data, and this difference is to some extent predictive of downstream performance (see table).

See for yourself if you can tell apart the different filtered datasets based on their quality (answers are at the end of this report):

1. link
2. link
3. link
4. link

We trained 1.4B models with x = {10,20} using default_train. We found that training on the top 20% of documents according to our BERT classifier did the best (e.g., 0.015 gain over fastText on Paloma bits/byte). If we had started from a larger pool, using the top 10% would have likely done better. See the table below (reporting bits/byte) and the Wandb report for details. We've been using bits/byte on Paloma as our main litmus test, and so the gains there are encouraging. But also of particular note are the very large gains we see on the Dolma "100 programming languages" subset (~0.1!), despite there not being an obvious connection between the high-quality examples we were using to train our quality filters (i.e., overall MMLU) and programming specifically.

	paloma	paloma/c4_en	paloma/dolma_100_programming_languages
FastText (10%)	1.37163	0.96626	1.21047
BERT (10%)	1.37512	0.96607	1.19601
FastText (20%)	1.36846	0.95703	1.16534
BERT (20%)	1.34925	0.95767	1.07757

Next steps:

Most models (including quality classifiers) must strike a balance between performance and compute. This experiment suggests there are gains to be had from expending more compute on quality filtering (i.e., using BERT instead of fastText), which motivates exploring more general resource allocation strategies for quality filtering (e.g., ensembling different filters or filtering in multiple stages).

Answers:

1. fastText (1%)
2. BERT (20%)
3. BERT (1%)
4. fastText (20%)