Fix wordpiece indexer truncation

[maksymbevza]

The problem

Sometimes offsets contain an extra item, because of the wrong condition in the if check.

What happens

Let's consider the case of _truncate_long_sequences = True with one start=[CLS] and end=[SEP] token, max_pieces == 4 and window_length == 2.

Let's say that our sentence is being encoded into following subtokens:
["the", "quick", "##est"]

For this case ["the"] will fit without any cut, but ["quick", "##est"] will not. The old code produced following ending offsets: [1, 3], but wordpieces after the cut are the following ["[CLS]", "the", "quick", "[SEP]"] and position 3 is invalid in this case cause it points to "[SEP]". But sometimes it also fails with index out of bound error, when offset goes beyond the size of the tensor.

The fix

By adding len(token) - 1 to current offset we make sure that the last wordpiece of the current token will fit in the sentence cut on line 208

[joelgrus]

thanks for this, I think this looks good, but can you

• give the tests more descriptive names, and
• add some comments to them

so that it's more obvious what behavior exactly they're testing and what would cause them to fail?

[maksymbevza]

@joelgrus , I've added comments for the tests
PTAL

[matt-gardner]

@joelgrus, ping.

[maksymbevza]

@joelgrus Could you please take a look?

[joelgrus]

looks good, thanks for the fix