Added a datatrove based pipeline for filtering tokenized data using scores.

[BlueCrescent]

• Included an example configuration file.
• Added datatrove and pydantic-settings to requirements.
• Note that modalities is also required for the pipeline to work, but it is not included in the requirements file.

[Copilot]

Pull Request Overview

This PR implements a data filtering pipeline using datatrove for filtering tokenized data based on scores. The pipeline processes JSONL files containing scores for data samples and filters corresponding tokenized datasets based on configurable thresholds.

• Adds a complete datatrove-based filtering pipeline with score parsing and data filtering components
• Introduces configuration management using pydantic-settings for both local and Slurm execution environments
• Updates dependencies to include datatrove and pydantic-settings

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/ml_filter/data_processing/score_based_filtering/step_score_parsing.py	Implements ScoresParser class for reading JSONL score files and mapping to tokenized data
src/ml_filter/data_processing/score_based_filtering/step_data_filtering.py	Implements DataFiltering class for filtering datasets based on score thresholds
src/ml_filter/data_processing/score_based_filtering/filter_pipeline.py	Main pipeline orchestration with configuration management and execution settings
pyproject.toml	Adds datatrove and pydantic-settings dependencies
configs/data_processing/example_filter_pipeline_config.yaml	Example configuration file for the filtering pipeline

Comments suppressed due to low confidence (1)

src/ml_filter/data_processing/score_based_filtering/filter_pipeline.py:241

• [nitpick] The error message could be more helpful by providing an example of how to use the FilterPipelineBuilder class directly or where to find documentation.

            "and use the FilterPipelineBuilder class directly."

[Copilot]

The FIXME comment indicates uncertainty about whether the 'account' parameter is supported. This should be resolved or documented properly rather than left as a FIXME in production code.

Suggested change

	account: str | None = None # FIXME is this supported?
	account: str | None = None # The Slurm account to charge for the job. Optional.