[LLM Runtime] Add Script for PPL Evaluation

[DDEle]

Type of Change: Feature

API not changed

Description

• Add PPL script
• Add cal_diff.py script for future CI @zhenwei-intel
• Fix UT (test_llm_runtime.py)

Expected Behavior & Potential Risk

N/A

How has this PR been tested?

Internal extension test:

• Llama2: https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension-WIP/32/artifact/report.html
• GPT-NEOX and OPT: https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension-WIP/33/artifact/report.html
• All models: https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension-WIP/34/

Dependency Change?

No

[airMeng]

add a note on homepage readme

[zhenwei-intel]

Do we have PPL data of pytorch and that of cpp graph fp32/int4?

[DDEle]

Do we have PPL data of pytorch and that of cpp graph fp32/int4?

https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension-WIP/32/artifact/report.html

[DDEle]

gpt-neox-20b and opt-1.3b checked in this internal extension test: https://inteltf-jenk.sh.intel.com/job/ITREX-cpp-graph-extension-WIP/33/artifact/report.html

[a32543254]

LGTM

[hshen14]

What's the purpose to introduce ppl script here? Can we just reuse lm-eval-harness to measure the ppl?

[DDEle]

What's the purpose to introduce ppl script here? Can we just reuse lm-eval-harness to measure the ppl?

It is for internal usage of LLM Runtime to keep track of the inference accuracy with minimal test time addition. It also works as a usage example of the LLM Runtime (llama.cpp also has a ppl script in its examples directory). We will investigate lm-eval-harness and see if it can help.