ValueError bug in greedy_until "too many values to unpack"

[danielfleischer]

Some tokenizers (Llama, Alpaca) return more than one token for "\n" even with add_special_tokens=False. It causes a value error in:

lm-evaluation-harness/lm_eval/base.py

Line 420 in 72b7f0c

(primary_until,) = self.tok_encode(until[0])

One can replace it with

primary_until = self.tok_encode(until[0])[0]

[haileyschoelkopf]

Thanks for raising this, and apologies for the bug!

In our upcoming version release, we handle multi-token stop sequences in a more principled+unified way (see here).

I've patched this + added a warning for the hf-causal model in #628 , and confirmed it doesn't crop up in the hf-causal-experimental case.