Model 1 expansion: translating to the null word

[bennokr]

This code looks for the words that probably make a sentence shorter, so you can delete them when translating:

c = Counter()
for s,t in corpus():
    sc = (len(s)-len(t)) / float(len(s))
    shorten = Counter(s)
    for w in shorten:
        c[w] += shorten[w] * sc
c.most_common()

The first three look good:

('te', 4.471595928036059),
(''', 4.035303262272552),
('s', 3.6778059597504105),

but the rest is not really.

What you'd actually want to do is look for bigrams that get translated into one word, and choose the one both words point to. Or, when translating, merge doubles:

Tot slot, blah blah -> Finally finally, blah blah -> Finally, blah blah

Another option is to add null words to the sentence pairs to make 'em longer:

def lengthen(pairs):
    for i in xrange(len(pairs)):
        fs,es = pairs[i]
        diff = len(fs) - len(es)
        if diff > 0:
            pairs[i] = (fs, es+['NULL']*diff)
        if diff < 0:
            pairs[i] = (fs+['NULL']*abs(diff), es)
        fs,es = pairs[i]
    return pairs