Script to build FastText training file from OLMo sources

[rauthur]

This is intended for use with the existing ft_tagger.py. Example invocation:

python -m ai2_llm_filters.core_tools.ft_dataset \
  -t /Users/russell/Projects/ai2-llm/tmp-data \
  -s /Users/russell/Projects/ai2-llm/tmp-data-neg /Users/russell/Projects/ai2-llm/tmp-data-neg-2 \
  -m sentence \
  --newlines skip \
  -o ./test-output.txt \
  --n-segments 3 \
  --pos-label pos \
  --neg-label neg

This will create a datafile that has 3 positive and 3 negative examples at the sentence level taken from all files under the tmp-data (pos examples) and tmp-data-neg(-2) folders. Locations for input and output can be on S3. Downloading and processing files is in parallel. More than one path can be specified for the negative examples (if this is useful it can apply to pos as well easily).

The above might output a training file like this:

__label__pos The Trofeo S.A.R. Princesa Sofia was born in 1968 with only one participating class, the Dragon, based at Real Club Náutico de Palma.
__label__pos It was then that the event became organised by four clubs: Palma, S'Arenal, San Antonio de la Playa and de Mar -instead of the present organiser Calanova-.
__label__pos Two years later the Olympic character of the event began to show with the temporary replacement of the Dragon by the Star.
__label__neg I like dogs
__label__neg I like cats
__label__neg I don't like alligators

Note that examples are not currently shuffled between positive and negative classes. Within the order is semi-random across files depending on multiprocessing speed per process.

[kyleclo]

lgtm; can u add a README walking through use of this to create a filter using C4?

[soldni]

neat stuff! okay to merge