Feature: allow evaluator to accept tokenized input

[jessecambon]

I believe the evaluator currently only supports text input. It would be useful if it could also accept tokenized input data (ie. if the data is already stored in tokenized format or we want to isolate the inference process from the tokenization process).

[jmwoloso]

+1

This would really be beneficial when the quantity and volume of text makes it impractical to use conventional methods for inference (e.g. tokenizing on the fly, payload size if inferencing via api endpoint, etc.)

[fxmarty]

Hello,

Thank you for trying out the evaluator! For which task is your use case? If it is text classification or question answering and you are using a transformers model, my first idea is that you could try the following:

class CustomTextClassificationPipeline(TextClassificationPipeline):
    def preprocess(self, inputs, **tokenizer_kwargs):
        return inputs

pipe = CustomTextClassificationPipeline(model="path_to_model")

and feed this pipeline (that skips preprocessing i.e. here tokenizing) to the evaluator. The result of this is that you can provide in the input_column argument of the evaluator compute() a column of a dataset where the tokenized inputs are stored.

Just to note, here is the original preprocess for the TextClassificationPipeline: https://github.com/huggingface/transformers/blob/a8e279579bbe7010d26e469c19636dd7142b1da5/src/transformers/pipelines/text_classification.py#L147-L162

For token classification, I think this would be more tricky. The issue is that the expected format of TokenClassificationEvaluator is more rigid because we must map back the predictions of the pipeline (that are on tokens) to the labels (that are on words). Therefore if you feed directly tokenized inputs, I am not sure how you can do the mapping.

[jmwoloso]

hi @fxmarty, yes this is for text classification. thank you for the suggestion to extend TextClassficiationPipeline, that seems like a solid idea (similar to extending the PT DataModule class).

[jessecambon]

@fxmarty, thanks this is a good approach. The custom pipeline does work for model inference on tokenized text. However, I did have to add an __init__ definition (see below) to get the evaluator to accept it.

The code below runs successfully until the final step where evaluator.compute() is called. On that command I this error

in _forward
return self.model(**model_inputs)
TypeError: ORTModelForSequenceClassification object argument after ** must be a mapping, not Tensor

However, I believe this is may be related to a separate issue with my dataset tokenization not generating the correct tensor shapes. As far as evaluate is involved, the tokenized data does appear to be getting passed to model so I would think this example will work once I resolve my token tensor shape issues.

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline
from transformers.pipelines import TextClassificationPipeline
from datasets import load_dataset
from evaluate import evaluator
eval = evaluator("text-classification")

model_arch="lvwerra/distilbert-imdb"

class CustomTextClassificationPipeline(TextClassificationPipeline):
    """
    Pipeline that accepts tokenized data as an input
    """
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.task='text-classification'

    def preprocess(self, inputs, **tokenizer_kwargs):
        return inputs

raw_data = load_dataset("imdb", split='test').shuffle(seed=42).select(range(10))

# grab first row of text
example_text=raw_data['text'][0]
print(f"example_text: {example_text}")

model=ORTModelForSequenceClassification.from_pretrained(model_arch,from_transformers=True)
tokenizer=AutoTokenizer.from_pretrained(model_arch)

# pipeline that takes tokenized text as an input
custom_pipe = CustomTextClassificationPipeline(model=model)

# pipeline that takes raw text as an input
standard_pipe = pipeline('text-classification',model=model,tokenizer=tokenizer)

example_text_tokenized=tokenizer(example_text,return_tensors='pt')

# test both pipelines
print(custom_pipe(example_text_tokenized))
print(standard_pipe(example_text))

# tokenized a dataset and pass it to the evaluator -------------------------------

data_tokenized = raw_data.map(
    lambda x: tokenizer(x["text"]),
    remove_columns=["text"]
)
data_tokenized=data_tokenized.rename_column("label", "labels") # huggingface model object expects 'labels'
data_tokenized.set_format("torch")

eval.compute(
    model_or_pipeline=custom_pipe,
    data=data_tokenized,
    input_column='input_ids',
    label_column='labels'
)

[fxmarty]

Hello, here is a corrected code snippet that does work, the forward expects a dict and not a tensor. Feels very hacky though, we should allow more flexibility in the input to the evaluator.

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline
from transformers.pipelines import TextClassificationPipeline
from datasets import load_dataset
from evaluate import evaluator
import torch
eval = evaluator("text-classification")

model_arch="lvwerra/distilbert-imdb"

class CustomTextClassificationPipeline(TextClassificationPipeline):
    """
    Pipeline that accepts tokenized data as an input
    """
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.task='text-classification'
        
    def preprocess(self, inputs, **tokenizer_kwargs):
        
        # `inputs` expected to be a tensor of input_ids, without batch dimension
        return {"input_ids": inputs.unsqueeze(0), "attention_mask": torch.ones((1,) + inputs.shape, dtype=torch.int64)}

raw_data = load_dataset("imdb", split='test').shuffle(seed=42).select(range(10))

# grab first row of text
example_text=raw_data['text'][0]
print(f"example_text: {example_text}")

##
model = ORTModelForSequenceClassification.from_pretrained(model_arch, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_arch)
##
# pipeline that takes tokenized text as an input
custom_pipe = CustomTextClassificationPipeline(model=model)

# pipeline that takes raw text as an input
standard_pipe = pipeline('text-classification',model=model,tokenizer=tokenizer)

example_text_tokenized = tokenizer(example_text,return_tensors='pt')

## test both pipelines
print(custom_pipe(example_text_tokenized["input_ids"][0]))
print(standard_pipe(example_text))

## tokenized a dataset and pass it to the evaluator -------------------------------

data_tokenized = raw_data.map(
    lambda x: tokenizer(x["text"]),
    remove_columns=["text"]
)
data_tokenized=data_tokenized.rename_column("label", "labels") # huggingface model object expects 'labels'
data_tokenized.set_format("torch")

eval.compute(
    model_or_pipeline=custom_pipe,
    data=data_tokenized,
    input_column='input_ids',
    label_column='labels',
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)

Note that the evaluator has no support for running evaluation on multi-GPU.

[lvwerra]

Thanks @fxmarty for the example. Closing this for now.