Tagger: use unnormalized probabilities for inference

[danieldk]

Description

Using unnormalized softmax avoids use of the relatively expensive exp function,
which can significantly speed up non-transformer models (e.g. I got a speedup
of 27% on a German tagging + parsing pipeline).

Types of change

Performance improvement

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

Draft since this requires explosion/thinc#583 and a new Thinc version.

[adrianeboyd]

There are users who use the scores directly and (I assume) would be expecting them to be normalized. I'm not saying we necessarily shouldn't do this, but this proposal doesn't give these users any way to control this behavior, does it?

[danieldk]

There are users who use the scores directly and (I assume) would be expecting them to be normalized. I'm not saying we necessarily shouldn't do this, but this proposal doesn't give these users any way to control this behavior, does it?

We could make this configurable. Maybe we should also only provide this functionality in a Tagger.v2 so that if anyone uses the probabilities it doesn't break for them (and they could opt-in to normalized probabilities with Tagger.v2)?

[adrianeboyd]

That sounds like a reasonable proposal.

[svlandeg]

Agreed that it's a good idea to make this configurable in a new version of the Tagger.

I find it slightly unintuitive to go from a v1 with always normalization, to a v2 with default normalization off. It might still catch some users off-guard that the behaviour changes between the two versions when you rely on defaults. But then again it is documented in the docs, and I can see why this would be beneficial in most cases / for most users. So I'm leaning towards keeping it like the PR has it currently :-)