Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Normalize all PTB produced tokens, not just the German ones, using NFC
Testing on 0.1% of Wikipedia (from a few years ago), this slows down the English tokenizer by about 1.5% The German umlaut unit test still works as well
- Loading branch information
1 parent
58a2288
commit d46fecd
Showing
3 changed files
with
4 additions
and
61 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters