Cascading Quality Filters

[BabyChouSr]

Description

Cascading filters - since we know most of the data is not good, getting the Llama-70B to label documents that are all pretty bad leads to pretty unbalanced classes when training the classifier. A confusion matrix of results show that 80% of labels are 0, I wonder how much it is actually learning to discriminate. Maybe if we have some initial filtering and then have it label those documents, it could lead to more balanced results.

Hypothesis or Goal

Use a two-step process to filter out data will lead to higher precision labels.

Links

(Delete any that aren't applicable)

• WandB Report: (link)
• Data Browser: (link)
• Experiment JSON: (link)
• (etc.)

Results

(What did you find, including relevant evaluation metrics, etc.)

[RohithKuditipudi]

also looking into this. concretely, here's what I have in mind: run multiple iterations of filtering to obtain final set of consolidated documents, where in each iteration we:

1. sample a seed set of documents out of the filtered pool from the previous iteration
2. score/label these seed documents using MEDU/Llama-70B.
3. train a BERT classifier (or if scores, regression) model on the labeled seed set
4. filter top x% (e.g., x = 50) of documents based on BERT model

[RohithKuditipudi]

My hypothesis is that this won't help too much for filtering e.g. top 20% of documents, but it will help in cases where we want a small set of very high quality documents (e.g., for cooldown mix). Like concretely, let's say we want the top 0.5% of documents (which should ideally be all 5s). Learning to distinguish 3s from 1s is a much different task from distinguishing 5s from 4s.

Another benefit of filtering iteratively is that inference costs go down for later steps. So we can use larger/more powerful classifiers for later iterations (e.g., even a small Llama-based classifier)

[percyliang]

What's the status?

[BabyChouSr]

I think we can mark this proof of concept complete since the proof of concept #1135 is complete and shows that the finemath scores perform better.

[dlwh]

This experiment has been archived to reduce clutter and preserve velocity (see PR #1999). It is last available in the archive/20251114 tag at https://github.com/marin-community/marin/tree/archive/20251114/experiments/exp963_cascade_finemath.py. Please open an issue if you need help unarchiving it.