Support Multi-choice Tasks (Manipulate the completion tokens)

[MM-IR]

Hi, I have noticed my previous failures now.
#Previous Failures

That is, for some multi-choice tasks, e.g., multi-choice sentence completions, in most common-sense reasoning tasks, we do need to manipulate the completion tokens, instead of current Samplingparams which generate token-by-token on LLMs themselves.

That is, autoregressive LMs will adjust their logprobs each step with different inputs. For MC tasks, we do need to input the correct prefix to LMs.

How can we enable so?

[MM-IR]

Otherwise, I do need to query multiple prompts for the same LM in serving this task.

For instance,

• Prompt: ABC.
• Completion 1: DE.
• Completion 2: FG.

Then, in current implementations, I can only prompt with ABC=>ABCD, and ABC=>ABCF for the cumulative probabilities results...

• Is this one less efficient, or achieving on par efficiency?

[MM-IR]

A better solution is to prompt query + Option together, and obtain their logits/logprobs at the same time..

[hmellor]

Closing because this is now supported via guided decoding #2819