ModuleNotFoundError: No module named 'lm_eval.api'

[yhyu13]

Hi,

I tried pip install lm_eval, but that package has no api module?

Which lm_eval should I install then?

MODEL_NAME=mamba-370m
MODEL_PATH=/media/hangyu5/Home/Documents/Hugging-Face/$MODEL_NAME
python evals/lm_harness_eval.py \
    --model mamba \
    --model_args pretrained=$MODEL_PATH \
    --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande \
    --device cuda \
    --batch_size 64 \
    tee lm_eval.txt

Traceback (most recent call last):
  File "/home/hangyu5/Documents/Git-repoMy/AIResearchVault/repo/Architecture/mamba/evals/lm_harness_eval.py", line 8, in <module>
    from lm_eval.api.model import LM
ModuleNotFoundError: No module named 'lm_eval.api'

Thanks!

[yhyu13]

Oh, I missed the submodule in 3rdParty

[yhyu13]

Could you refine README a little bit, I did not find there was a submodule there

git submodule update --init && \
cd 3rdparty/lm-evaluation-harness && \
pip install -r requirements.txt && pip install -e .

[tridao]

This was in the doc for Evaluation, did that not work?