What
Failed to evaluate "truthfulqa" benchmark using lm_eval package. The benchmark needs generate method at least.
- To make transformers
generate (from GenerateMixin) usable (without kv-cache it's very slow) we need to support DynamicCache from transformers (right now it's just a list of kv-tuples).
- Or we need to reimplement
generate.
What
Failed to evaluate "truthfulqa" benchmark using
lm_evalpackage. The benchmark needsgeneratemethod at least.generate(fromGenerateMixin) usable (without kv-cache it's very slow) we need to supportDynamicCachefromtransformers(right now it's just a list of kv-tuples).generate.