non-breaking space does not appear in token stream

[retorquere]

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );

const text = 'Hello\u00a0World';
const doc = nlp.readDoc(text);

doc.sentences().each(sentence => {
  sentence.tokens().each(token => {
    console.log([token.out(nlp.its.precedingSpaces), token.out()])
  })
})

does not return \u00a0 as either precedingSpaces or token.out()

[rachnachakraborty]

Hi @retorquere

Looks like unicode handling needs a rapid solution.

We will take it up as soon as possible.

Appreciate your inputs, please keep posting as and how you uncover such cases.

Best,
Rachna

[sanjayaksaxena]

Published new versions with the feature to handle nbsps:

1. wink-nlp version 2.2.0
2. wink-eng-lite-web-model 1.6.0

Thanks @retorquere for your contribution.

[retorquere]

Cool, thanks. Is it true there's a way to use wink for tokenization only (and get a faster and/or smaller solution that way)? I thought I saw that somewhere, would like to test that.

[sanjayaksaxena]

Yes! You can do it by passing an empty pipe during model instantiation:

nlp = winkNLP( model, [] )

This should give you ~4x improvement in the speed.

You can read more about it at Processing Pipeline.

[retorquere]

Will this also bring down the bundle size? My previous attempt gave me a 15Mb bundle where compromise/one was about a tenth of that.

[sanjayaksaxena]

Bundle size will not reduce. But the contribution of winkNLP+model should be less than 4MB to the total bundle size using browserify.

[retorquere]

I'm using esbuild; so 4 MB include the model? Then I've been doing something wrong. But I'm afraid I'm pulling this issue off-topic.

[sanjayaksaxena]

Yes the contribution of winkNLP + model together should contribute <4MB. Let us use email wink@graype.in or open separate thread for any further discussions which are off-topic.

[retorquere]

I found my mistake and the bundle size is as you indicate. Thanks for your help!