a initial version of executor

[zhaochenyang20]

Description

Following the document string, I implement the initial version of ModelExcuctor. However, I have some quick question:

1. How can we get the "confidence", and where will we use it? Currently, I am just using this as the confidence:

            logits = output[0].float()
            probs = torch.softmax(logits, dim=-1)
            confidence = probs.mean().item()

I think the calculation probs.mean().item() does provide a measure that can be loosely interpreted as the model's confidence in generating the output, but not enough.

The softmax probabilities obtained from the model's output can be seen as an indication of the model's belief or preference for each token in the generated sequence. By taking the average of these probabilities, probs.mean().item() provides a rough estimate of the overall confidence or likelihood of the generated output sequence.

2. What is auxiliary_info, where will we use it, and how to get it?

References

• NA

Blocked by

• debug tokenizer #60

[viswavi]

Seems like we should support batch execution? This can be done in a separate PR if you like

[viswavi]

Looks good! Only have a few small points for revisino

[viswavi]

Should we verify that the model in the else case is a transformers.AutoModelForCausalLM?

Suggested change

	else:
	output = model.generate(input_ids=encoded_input["input_ids"])
	elif issubclass(model.__class__, transformers.AutoModelForCausalLM):
	output = model.generate(input_ids=encoded_input["input_ids"])
	else:
	raise ValueError(f"Model class {model.__class__} not supported")

[zhaochenyang20]

Hmmm. I agree with you and actually tried this before. But issubclass(model.__class__, transformers.AutoModelForCausalLM) would fail even for gpt2. 🤔

[zhaochenyang20]

In [3]:     gpt2_model_name = "gpt2"
   ...:     gpt2_model = AutoModelForCausalLM.from_pretrained(gpt2_model_name)
   ...:     gpt2_tokenizer = AutoTokenizer.from_pretrained(gpt2_model_name)

In [4]: model = gpt2_model

In [5]: issubclass(model.__class__, transformers.AutoModelForCausalLM)
Out[5]: False

[zhaochenyang20]

Actually, the inherited tree of GPT-2 is:

nn.Module -> PreTrainedModel -> GPT2PreTrainedModel -> GPT2LMHeadModel.

So issubclass(model.__class__, transformers.AutoModelForCausalLM) would fail.

But do we have better methods of type check? 🤔

[viswavi]

ah I see....if there's no general type that can cover all autoregressive LM models then I think it's fine how you have it now

[viswavi]

It's a bit weird to have this setup done in the test. Can you explain this a little more?

[zhaochenyang20]

Just as what happens in #60. Directly use gpt2_tokenizer is not enough.

Using pad_token, but it is not set yet.
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

[viswavi]

ah I see....if there's no general type that can cover all autoregressive LM models then I think it's fine how you have it now