[quantization] Introduce a script for LLM evaluation

[stamalakhov]

This PR introduces an option to run many LLM-related tasks using lm_eval package.

To use it please make sure that you have installed lm_eval

pip install lm-eval

The PR gives a chance to range quantization results not only using PPL degradation but also using accuracy on a list of benchmarks.

python tico/quantization/evaluation/script/llm_tasks_eval.py --model "HuggingFaceTB/SmolLM2-135M-Instruct"

Loading FP model …
`pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2376/2376 [00:27<00:00, 85.22it/s]
Running loglikelihood requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9501/9501 [06:03<00:00, 26.17it/s]
results of HuggingFaceTB/SmolLM2-135M-Instruct evaluation:
| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_easy|      1|none  |     0|acc     |↑  |0.5400|±  |0.0102|
|        |       |none  |     0|acc_norm|↑  |0.4882|±  |0.0103|

Draft: #436
TICO-DCO-1.0-Signed-off-by: s.malakhov s.malakhov@partner.samsung.com

[mhs4670go]

FYI, there is a visualization API.

from lm_eval.utils import make_table

results = lm_eval.simple_evaluate(...)
print(make_table(results))

[stamalakhov]

Wow. Thank you! I didn't know that.

[stamalakhov]

@mhs4670go I'll update the script.

[mhs4670go]

LGTM

[Torrero]

Overall LGTM,
But, could you provide your opinion, please, maybe lm_eval should be added to dependencies of the project

[mhs4670go]

I don't think we would add a dependency on lm_eval. That is why this script is located at just script folder and has no tests.

When we need external dependencies like transformers or lm_eval, it would be good to have dedicated scripts or workflows in internal repo.

[mhs4670go]

LGTM