Fix alignment algorithm

[tamuhey]

From #87

FIx the below method with pytokenizations, which is I created for this purpose.

spacy-transformers/spacy_transformers/pipeline/wordpiecer.py

Line 108 in 378d6aa

def _align(self, segment, wp_tokens, *, offset=0):

This library is based on "shortest edit script" and can handle noisy tokenizations, for example:

a = ["げん", "ご"]
b = ["けんこ"] # all accents are dropped (が -> か, ご -> こ)
a2b = [[0], [0]]
b2a = [[0, 1]]
assert tokenizations.get_alignments(a, b) == (a2b, b2a)

(I thinks this library is useful for spacy.align function. see explosion/spaCy#4554)

[tamuhey]

Passed https://github.com/explosion/spacy-transformers/blob/master/examples/test_wordpiece_alignment.py

[tamuhey]

I created PR in spacy: explosion/spaCy#4878