Skip to content
Allo-media data selection tool
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Allo-media data selection tool

This code implements the data selection method and algorithms proposed in Axelrod's paper CYNICAL SELECTION OF LANGUAGE MODEL TRAINING DATA, based on the paper's explanations and the Perl implementation Axelrod proposed on github

Comments in code and details on usage to come, but it's pretty simple right now.

Basic usage

Say you have a (small) representative corpus (task.txt) and a (big) general one (unadapted.txt) and you want to select sentences from the big corpus that look like the small corpus ones.

Usage would be:

./ --task task.txt --unadapted unadapted.txt

This will produce a .jaded file containing the selected sentences using the following tab-separated format:

model score sentence score (penalty + gain) length penalty sentence gain sentence id (in the selection) sentence id (in the unadapted corpus) best word word gain sentence.

See header of the script for available options, here is the two most important:

batch: essential with big corpora, allows to select more than one sentence at a time, see Axelrod's paper

iterate: iterate selection runs till no more than 10% of original size can be removed

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.