Using fast tokenizers with pipelines

[AlecS12]

🚀 Feature request

Currently tokenizers are not working with QA pipeline, because they do not have the tokenize method implemented. Speeding up the tokenization would be really beneficial for my application.

Motivation

Your contribution

[mfuntowicz]

Hi @AlecS12,

I'm currently working on integrating tokenizers library inside transformers with pipelines support.

It should not be long now before it lang on master / get released.

You can track the development here: #2674
If you want to checkout out the branch tokenizers-v2 and give it a try, I'm more than happy to get your feedback.

Morgan

[AlecS12]

Hi @mfuntowicz,

That's great news. I checked out tokenizers-v2 and tried it in a web server (flask) and jupyterlab. In both cases got the same error. Could you please look into this?

from transformers import pipeline
nlp = pipeline('question-answering',  model='bert-large-uncased-whole-word-masking-finetuned-squad')
nlp({
    'question': 'Where is the cookie?',
    'context': 'I keep cookies in a red plastic container.'
})

...
I0212 15:15:53.546577 140155558979392 modeling_utils.py:456] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin from cache at /home/a652726/.cache/torch/transformers/ca2ac20761877486c1e2204d99653106b9adacf9a5eb18ec71b41d2dbef42103.2db7ae79c41a184c87600faabafa1369db2b16457723fd154ca3b436c4172807
convert squad examples to features:   0%|          | 0/1 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/data/home/a652726/transformers/src/transformers/data/processors/squad.py", line 141, in squad_convert_example_to_features
    truncation_strategy="only_second" if tokenizer.padding_side == "right" else "only_first",
  File "/data/home/a652726/transformers/src/transformers/tokenization_utils.py", line 1741, in encode_plus
    **kwargs,
  File "/data/home/a652726/transformers/src/transformers/tokenization_utils.py", line 1676, in batch_encode_plus
    tokens = self._tokenizer.encode(*batch_text_or_text_pairs[0])
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py", line 131, in encode
    return self._tokenizer.encode(sequence, pair)
TypeError
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
<ipython-input-1-c0bdf7f90854> in <module>
      3 nlp({
      4     'question': 'Where is the cookie?',
----> 5     'context': 'I keep cookies in the red plastic container.'
      6 })

...

[mfuntowicz]

I will definitively have a look, and will keep you posted.

Thanks for reporting

[AlecS12]

Hi @mfuntowicz,

I installed the latest 2.5.1 release and the pipeline error is still there. Had to roll back to 2.4.1.

[AlecS12]

release 2.5.1 does not produce the error by default anymore, because it changed the default Autotokenizer to slow, but the bug is still there:

import transformers
from transformers import pipeline
tokenizer = transformers.AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)
nlp = pipeline('question-answering',  model='bert-large-uncased-whole-word-masking-finetuned-squad', tokenizer=tokenizer)
nlp({
    'question': 'Where is the cookie?',
    'context': 'I keep cookies in the red plastic container.'
})

nlp({
    'question': 'Where is the cookie?',
    'context': 'I keep cookies in the red plastic container.'
})


convert squad examples to features:   0%|          | 0/1 [00:00<?, ?it/s]W0226 10:50:27.573424 140277524375360 tokenization_utils.py:1782] Fast tokenizers add special tokens by default. To remove special tokens, please specify`add_special_tokens=False` during the initialisation rather than when calling `encode`,`encode_plus` or `batch_encode_plus`.
W0226 10:50:27.576760 140277524375360 tokenization_utils.py:1782] Fast tokenizers add special tokens by default. To remove special tokens, please specify`add_special_tokens=False` during the initialisation rather than when calling `encode`,`encode_plus` or `batch_encode_plus`.
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/data/home/a652726/transformers/src/transformers/data/processors/squad.py", line 141, in squad_convert_example_to_features
    truncation_strategy="only_second" if tokenizer.padding_side == "right" else "only_first",
  File "/data/home/a652726/transformers/src/transformers/tokenization_utils.py", line 1889, in encode_plus
    **kwargs,
  File "/data/home/a652726/transformers/src/transformers/tokenization_utils.py", line 1815, in batch_encode_plus
    tokens = self._tokenizer.encode(*batch_text_or_text_pairs[0])
  File "/home/a652726/miniconda3/envs/nlp2/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py", line 141, in encode
    return self._tokenizer.encode(sequence, pair)
TypeError
"""

'''

[mfuntowicz]

Hi @AlecS12,

Thanks for trying out 2.5.1. The issue is still there because for the question-answering pipeline we're relying on a method from the squad data processor squad_convert_example_to_feature which is not compatible which the fast tokenizers.

I'll have soon have a look at this to make it compatible with the fast tokenizers.

Sorry for the inconvenience.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AlecS12]

Hi @mfuntowicz,,

The problem is still there in 2.10.1. Could you please reopen the issue and fix it?