[Feature] Support is_split_into_words in the TokenClassificationPipeline.

[yushi2006]

What does this PR do?

Fixes #30757

This PR solves the problem that when using an already tokenized input in the TokenClassificationPipeline by adding a tokenizer_parameter called is_split_into_words. I think we will need the aggregation code because some tokenizers are subword tokenizers, I have solved the problem that start and end was not correctly assigned when using the pipeline by reconstructing the sentence and create a map that stores the position of each word and pass it along with words_id to the postprocess and in gather_pre_entities, we use the map and words_id to produce the final position.

from transformers import pipeline
from datasets import load_dataset

dataset = load_dataset("conll2003", split="test[:5]")

ner_pipeline = pipeline("token-classification", model="dslim/bert-base-NER", aggregation_strategy="first")

for example in dataset:
    tokens = example['tokens']

    predictions = ner_pipeline(tokens, is_split_into_words=True)

    for entity in predictions:
        print(entity)

Now this example doing inference on a batch of a preprocessed dataset. it gives this output.

[{'entity_group': 'MISC', 'score': np.float32(0.48014355), 'word': 'JAPAN', 'start': 9, 'end': 14}, {'entity_group': 'PER', 'score': np.float32(0.43156916), 'word': 'LUCKY', 'start': 19, 'end': 24}, {'entity_group': 'ORG', 'score': np.float32(0.64400136), 'word': 'CHINA', 'start': 31, 'end': 36}]
[{'entity_group': 'PER', 'score': np.float32(0.9979421), 'word': 'Nadim Ladki', 'start': 0, 'end': 11}]
[{'entity_group': 'LOC', 'score': np.float32(0.9851431), 'word': 'AL - AIN', 'start': 0, 'end': 6}, {'entity_group': 'LOC', 'score': np.float32(0.99941224), 'word': 'United Arab Emirates', 'start': 9, 'end': 29}]
[{'entity_group': 'LOC', 'score': np.float32(0.9998572), 'word': 'Japan', 'start': 0, 'end': 5}, {'entity_group': 'MISC', 'score': np.float32(0.99814206), 'word': 'Asian Cup', 'start': 33, 'end': 42}, {'entity_group': 'LOC', 'score': np.float32(0.9997607), 'word': 'Syria', 'start': 78, 'end': 83}, {'entity_group': 'MISC', 'score': np.float32(0.77703416), 'word': 'Group C', 'start': 89, 'end': 96}]
[{'entity_group': 'LOC', 'score': np.float32(0.9998673), 'word': 'China', 'start': 4, 'end': 9}, {'entity_group': 'LOC', 'score': np.float32(0.9997266), 'word': 'Uzbekistan', 'start': 119, 'end': 129}]

Who can review?

@Rocketknight1 @ArthurZucker

[Rocketknight1]

Overall looks good, but made a couple of comments!

[Rocketknight1]

The return type is the same in both cases so I'm not sure why we need the overload here!

[yushi2006]

You're totally right that overload was unnecessary. Not sure what I was thinking there 😅
The return type doesn’t change at all, so I’ve removed it from the code now.

[Rocketknight1]

Dumb question, but really is_split_into_words means that the input has been split into tokens rather than words, right? If any of the tokens aren't full words, this display will be wrong.

I'm not sure if that's fixable because different tokenizers use different ways of indicating word completions, I'm just not sure if this will always work perfectly!

[yushi2006]

Hey! Not a dumb question at all actually, a very insightful one.

You're right to question the assumptions around is_split_into_words, especially with the diversity in how tokenizers handle word boundaries. Luckily, this specific implementation accounts for that by checking whether a token is a subword using a simple but effective heuristic:

is_subword = len(word) != len(word_ref)

This allows us to detect subword tokens even when tokenizers split words in unpredictable ways. The aggregate_word function then groups these subword tokens together correctly, leveraging the fact that tokenizers provide enough information to reconstruct the original word groupings.

That said, your point did make me realize there’s a deeper edge case here languages like Japanese or Chinese that don’t use whitespace as word boundaries.

So while the current code handles most whitespace-separated languages well, you're absolutely right that a more general, tokenizer-agnostic approach would be much trickier and probably need deeper integration with a smarter word alignment logic.

One idea I had for this is to expose a preprocessing parameter — something like a delimiter — so the user can explicitly define how their original words were separated. For whitespace-based languages, it could default to ' ', but for others, users could pass an empty string or custom logic. For example:

sentence = delimiter.join(words)

Thanks for the question it helped me think more critically about generalization.

Let me know if you have any ideas or pointers on making this more robust I’d love to explore better ways to handle those cases.