Duplicity rows in nametag output

[matyaskopp]

There are duplicated rows for this input:

Směrnice Evropského parlamentu a Rady číslo 98/70 ES o jakosti benzinu a motorové nafty stanoví závazný cíl dosáhnout do roku 2020 6 % snížení emisí skleníkových plynů z pohonných hmot v porovnání s rokem 2010.

...skipping...

[matyaskopp]

missing category:

[foxik]

This seems like a bug in the decoder, will investigate.

[foxik]

We have an idea what is going on and have a plan how to fix it, but it might take some time before we get to it. Keeping open as a remainder until then.

[strakova]

Deployed server fix which prevents the decoder to go crazy and predict unlimited number of labels per token by imposing an upper limit on labels per token.

[matyaskopp]

thanks, @strakova.
but it can still produce duplicity rows:

or simpler example, that in the previous version of "crazy decoder", had not become so crazy:

[strakova]

Ahh, right. Thanks. Fixed only partially. Reopening.

[strakova]

Fixed (actively removing duplicities from the decoder output).