Optimize tokenizer.decode() Performance for List[int] Inputs

[n0gu-furiosa]

Feature request

When calling tokenizer.decode() with a List[int] as token_ids, the method appears to be significantly slower than necessary due to redundant to_py_obj conversions.

Motivation

Example:

import time
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('hf-internal-testing/llama-tokenizer')
token_ids = [869] * 2000

start = time.time()
for _ in range(1000):
    tok.decode(token_ids)
print(time.time() - start)

The trace results for the above code show that most of the time is spent on repeated to_py_obj calls, rather than in the actual _decode function:

In this case, since the input is already a List[int], passing it through to_py_obj seems redundant. By adding a conditional check to bypass this line for List[int] inputs:

transformers/src/transformers/tokenization_utils_base.py

Line 3868 in 6a26279

token_ids = to_py_obj(token_ids)

…the example code improves by nearly 10x in my environment (from ~7s to ~0.7s).

Your contribution

I wasn’t sure where the best place to apply this optimization would be—either within decode() or inside to_py_obj()—so I haven’t opened a PR yet. I’d be happy to contribute a fix if there’s guidance on where such a change would be most appropriate.

[Rocketknight1]

This is most likely caused by this section in to_py_obj():

if isinstance(obj, (dict, UserDict)):
    return {k: to_py_obj(v) for k, v in obj.items()}
elif isinstance(obj, (list, tuple)):
    return [to_py_obj(o) for o in obj]

When a flat list is passed in, to_py_obj() will be called on each element of the list, which means 2000 function calls are required in your test. If you can figure out an optimization in that function that retains correct output without that speed penalty, we'd definitely welcome a PR!

[n0gu-furiosa]

Thank you for your guidance @Rocketknight1! I've opened a PR for this.