Apply tokenizer truncation before post-processor #307
Conversation
| Tokenizer.set_truncation(tokenizer, | ||
| max_length: upper_bound_length, | ||
| direction: opts[:truncate_direction] | ||
| ) |
There was a problem hiding this comment.
When we apply the tokenizer it does:
- Truncate the sequence if configured (accounting for the number of tokens added by 2.)
- Applies post-processor (in particular templating that adds leading trailing tokens)
- Applies padding if configured
So for example, lets say we have sequence that tokenizes to [10, 20, 30, 40] and the processor that adds 0 at the beginning and 1 at the end. And let's say we want to enforce length 4. If we tokenize without truncation, it's going to give us [0, 10, 20, 30, 40, 1], and then we manually truncate to [0, 10, 20, 30], which looses the added end token. On the other hand if we configure truncation it's going to compute [10, 20, 30, 40], then first truncate to 4 - 2 addeditional tokens, so [10, 20], and then add the tokens [0, 10, 20, 1].
Now the issue is that Tokenizer.set_truncation copies the underlying tokenizer. It is relatively cheap (bumping some refcounts and copying added_vocabulary, not the full vocabulary). That said ideally we would configure the tokenizer once. So what we can do deprecate options like :length, :truncate_direction, :pad_direction, and instead have a separate function to configure that. @josevalim wdyt?
Closes #306.