Add tokenisation support for the Tetun language

[raphaelmerx]

Description

Adding support for tokenisation of the Tetun Dili language (code tdt). Tetun has words that contain apostrophes (e.g "I" in Tetun is "ha'u"). The logic here will keep apostrophes when they are part of words, but will tokenize them if they are used for quotes.

Test

echo "bainhira sa'e aviaun husi Austrália ba Timor-Leste dehan 'oh kadeira sa'e ona'" | perl mosesdecoder/scripts/tokenizer/tokenizer.perl -threads 1 -l tdt -no-escape
Tokenizer Version 1.1
Language: tdt
Number of threads: 1
bainhira sa'e aviaun husi Austrália ba Timor-Leste dehan ' oh kadeira sa'e ona '

[hieuhoang]

thanks!