NGrams doesnt support words with hyphen and slash in English

[sam68740]

There are a few words in English that contain hyphen or slash

Example:

• image-based
• text-based
• links/CTA

It would be great if Natural could manage these cases.

let text = "links text-based opposed image-based links/CTA’s"
var NGrams = natural.NGrams;
const T = natural.AggressiveTokenizer;
const tokenizer = new T();
NGrams.setTokenizer(tokenizer);
console.log(NGrams.ngrams(text, 1));

Output: [["links"], ["text"], ["based"], ["opposed"], ["image"], ["based"], ["links"], ["CTA"], ["s"]]

[Hugo-ter-Doest]

The root cause of this behaviour is the tokenizer. Will look into adapting the tokenizer to support these characters.

[sam68740]

There is also other issues in other language. For instance the Eszett symbol ß is also not supported. If you can also check, it will be nice. Thanks

…

On Mon, May 29, 2023 at 8:11 PM Hugo ter Doest ***@***.***> wrote: The root cause of this behaviour is the tokenizer. Will look into adapting the tokenizer to support these characters. — Reply to this email directly, view it on GitHub <#656 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHL44R6OWCQAPK5LD7FEUDXITRD7ANCNFSM6AAAAAARMJBVLY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Hugo-ter-Doest]

Solved in #706