Non-Breaking Prefixes are stripped

[pypae]

Just a note: In the original perl script, non-breaking prefixes are not stripped when loading them from the file.

In my opinion stripping non-breaking prefixes is wanted, because there can be some unwanted spaces that are very hard to spot in the nbp-files.

I actually found this issue because in nonbreaking_prefix.it on line 139 es is followed by a space, causing dots following es to be split up (using the perl implementation but not using this implementation)

[alvations]

Thanks for catching this!

I did a sanity check, seems like the Italian nonbreaking_prefix.it on sacremoses matches the original moses one.

import requests

moses = 'https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/share/nonbreaking_prefixes/nonbreaking_prefix.{}'
sacremoses = 'https://raw.githubusercontent.com/alvations/sacremoses/master/sacremoses/data/nonbreaking_prefixes/nonbreaking_prefix.{}'

langs = ['ca', 'cs', 'de', 'el', 'en', 'es', 'fi', 'fr', 
         'ga', 'hu', 'is', 'it', 'lv', 'nl', 'pl', 'pt', 
         'ro', 'ru', 'sk', 'sl', 'sv', 'ta']

for lang in langs:
    mo = requests.get(moses.format(lang)).content.decode('utf8')
    sm = requests.get(sacremoses.format(lang)).content.decode('utf8')
    print(lang, mo == sm)

[out]:

ca False
cs True
de True
el True
en True
es True
fi True
fr False
ga True
hu False
is True
it True
lv True
nl True
pl True
pt True
ro True
ru True
sk True
sl False
sv True
ta True

[alvations]

Resolved c.f. #59