Skip to content

Latest commit

 

History

History
307 lines (229 loc) · 26.3 KB

README.md

File metadata and controls

307 lines (229 loc) · 26.3 KB

JP Language Model Evaluation Harness

Leaderboard

model average jcommonsenseqa jnli marc_ja jsquad jaqket_v2 xlsum_ja xwinograd_ja mgsm eval script
stabilityai-japanese-stablelm-instruct-alpha-7b 54.71 82.22 52.05 82.88 63.26 74.83 7.79 72.68 2 models/stabilityai/stabilityai-japanese-stablelm-instruct-alpha-7b/harness.sh
stabilityai-japanese-stablelm-base-alpha-7b 51.06 33.42 43.34 96.73 70.62 78.09 10.65 72.78 2.8 models/stabilityai/stabilityai-japanese-stablelm-base-alpha-7b/harness.sh
rinna-bilingual-gpt-neox-4b-instruction-sft 47.75 49.51 47.08 95.28 55.99 61.17 5.51 64.65 2.8 models/rinna/rinna-bilingual-gpt-neox-4b-instruction-sft/harness.sh
rinna-bilingual-gpt-neox-4b-instruction-ppo 47.18 48.79 48.23 96.09 54.16 57.65 5.03 65.07 2.4 models/rinna/rinna-bilingual-gpt-neox-4b-instruction-ppo/harness.sh
llama2-13b-chat 47.02 72.56 35.62 59.92 67.69 48.2 15.14 63.82 13.2 models/llama2/llama2-13b-chat/harness.sh
llama2-13b 46.32 74.89 21.98 38.89 76.14 67.7 18.11 62.88 10 models/llama2/llama2-13b/harness.sh
rinna-japanese-gpt-neox-3.6b-instruction-ppo 46.32 44.06 54.19 89.61 51.62 50.95 6.63 69.13 4.4 models/rinna/rinna-japanese-gpt-neox-3.6b-instruction-ppo/harness.sh
rinna-japanese-gpt-neox-3.6b-instruction-sft-v2 45.23 40.57 53.45 89.88 44.91 52.84 6.14 71.22 2.8 models/rinna/rinna-japanese-gpt-neox-3.6b-instruction-sft-v2/harness.sh
rinna-japanese-gpt-neox-3.6b-instruction-sft 43.82 38.07 44.58 90.62 47.41 53.69 4.74 69.45 2 models/rinna/rinna-japanese-gpt-neox-3.6b-instruction-sft/harness.sh
llama2-7b 42.96 52.64 28.23 86.05 58.4 38.83 9.32 64.65 5.6 models/llama2/llama2-7b/harness.sh
rinna-japanese-gpt-neox-3.6b 41.79 31.64 34.43 74.82 47.91 68.38 5.16 70.8 1.2 models/rinna/rinna-japanese-gpt-neox-3.6b/harness.sh
llama2-7b-chat 41.31 55.59 29.54 90.41 59.34 17.96 2.34 66.11 9.2 models/llama2/llama2-7b-chat/harness.sh
rinna-bilingual-gpt-neox-4b 40.03 20.82 55.22 59.55 50.79 59.45 5.55 66.42 2.4 models/rinna/rinna-bilingual-gpt-neox-4b/harness.sh
cyberagent-open-calm-7b 38.8 24.22 37.63 74.12 45.79 60.74 2.04 65.07 0.8 models/cyberagent/cyberagent-open-calm-7b/harness.sh
cyberagent-open-calm-3b 38.61 27.79 40.35 86.21 40.45 46.91 1.95 63.61 1.6 models/cyberagent/cyberagent-open-calm-3b/harness.sh
rinna-japanese-gpt-1b 36.92 34.76 37.67 87.86 26.18 37.03 5.34 64.55 2 models/rinna/rinna-japanese-gpt-1b/harness.sh
rinna-japanese-gpt-neox-small 31.12 34.22 30.11 83.35 5.80 31.78 3.85 57.24 1.6 models/rinna/rinna-japanese-gpt-neox-small/harness.sh

How to evaluate your model

  1. git clone https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable

    git clone -b jp-stable https://github.com/Stability-AI/lm-evaluation-harness.git
    cd lm-evaluation-harness
    pip install -e ".[ja]"
  2. Choose your prompt template based on docs/prompt_templates.md

  3. Replace TEMPLATE to the version and change MODEL_PATH . And, save the script as harness.sh

    MODEL_ARGS="pretrained=MODEL_PATH"
    TASK="jsquad-1.1-TEMPLATE,jcommonsenseqa-1.1-TEMPLATE,jnli-1.1-TEMPLATE,marc_ja-1.1-TEMPLATE"
    python main.py \
        --model hf-causal \
        --model_args $MODEL_ARGS \
        --tasks $TASK \
        --num_fewshot "2,3,3,3" \
        --device "cuda" \
        --output_path "result.json"
  4. Run!

    sh harness.sh

We evaluated some open-sourced Japanese LMs. Pleasae refer to harness.sh inside models folder.

JP Tasks

For more details, please see docs/jptasks.md.

Tasks Supported Prompt Templates
JSQuAD 0.1 / 0.2 / 0.3 / 0.4
JCommonsenseQA 0.1 / 0.2 / 0.3 / 0.4
JNLI 0.2 / 0.3 / 0.4
MARC-ja 0.2 / 0.3 / 0.4
JaQuAD 0.1 / 0.2 / 0.3 / 0.4
JBLiMP -
XLSum-ja 0.0 / 0.3 / 0.4
JAQKET 0.1 / 0.2 / 0.3 / 0.4

Language Model Evaluation Harness

codecov

Overview

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

Install

To install lm-eval from the github repository main branch, run:

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

To install additional multilingual tokenization and text segmentation packages, you must install the package with the multilingual extra:

pip install -e ".[multilingual]"

To support loading GPTQ quantized models, install the package with the auto-gptq extra:

pip install gekko
pip install -e ".[auto-gptq]"

Basic Usage

Note: When reporting results from eval harness, please include the task versions (shown in results["versions"]) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the Task Versioning section for more info.

To evaluate a model hosted on the Hugging Face Hub (e.g. GPT-J-6B) on tasks with names matching the pattern lambada_* and hellaswag you can use the following command:

python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks lambada_*,hellaswag \
    --device cuda:0

Also check the script for running evalutation suites.

Additional arguments can be provided to the model constructor using the --model_args flag. Most notably, this supports the common practice of using the revisions feature on the Hub to store partially trained checkpoints:

python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000 \
    --tasks lambada_openai,hellaswag \
    --device cuda:0

To evaluate models that are loaded via AutoSeq2SeqLM in Hugging Face, you instead use hf-seq2seq. To evaluate (causal) models across multiple GPUs, use --model hf-causal-experimental

Warning: Choosing the wrong model may result in erroneous outputs despite not erroring.

To use with PEFT, take the call you would run to evaluate the base model and add ,peft=PATH to the model_args argument as shown below:

python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=EleutherAI/gpt-j-6b,peft=nomic-ai/gpt4all-j-lora \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0

Our library also supports the OpenAI API:

export OPENAI_API_SECRET_KEY=YOUR_KEY_HERE
python main.py \
    --model gpt3 \
    --model_args engine=davinci \
    --tasks lambada_openai,hellaswag

While this functionality is only officially maintained for the official OpenAI API, it tends to also work for other hosting services that use the same API such as goose.ai with minor modification. We also have an implementation for the TextSynth API, using --model textsynth.

To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the --check_integrity flag:

python main.py \
    --model gpt3 \
    --model_args engine=davinci \
    --tasks lambada_openai,hellaswag \
    --check_integrity

To evaluate mesh-transformer-jax models that are not available on HF, please invoke eval harness through this script.

💡 Tip: You can inspect what the LM inputs look like by running the following command:

python write_out.py \
    --tasks all_tasks \
    --num_fewshot 5 \
    --num_examples 10 \
    --output_base_path /path/to/output/folder

This will write out one text file for each task.

Evaluation Suites

If you have multiple tasks that you routinely run as an evaluation suite, you can save the suite configuration in a single file and run it with different models. Save a suite config to lm_eval/suites/configs/[suite].conf, formatted like this:

[tasks.my_task]
version = 1.0
fewshot = 2

[tasks.other_task]
version = 1.1
fewshot = 3

Then you can run the suite like this:

python scripts/run_suite.py [model_path] [suite_name] [prompt_version] -m [model_args]

For prompt versions, see the prompt docs and the list of prompt names.

Advanced Usage

For models loaded with the HuggingFace transformers library, any arguments provided via --model_args get passed to the relevant constructor directly. This means that anything you can do with AutoModel can be done with our library. For example, you can pass a local path via pretrained= or use models finetuned with PEFT by taking the call you would run to evaluate the base model and add ,peft=PATH to the model_args argument:

python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=EleutherAI/gpt-j-6b,peft=nomic-ai/gpt4all-j-lora \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0

GPTQ quantized models can be loaded by specifying their file names in ,quantized=NAME (or ,quantized=True for default names) in the model_args argument:

python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=model-name-or-path,quantized=model.safetensors,gptq_use_triton=True \
    --tasks hellaswag

We support wildcards in task names, for example you can run all of the machine-translated lambada tasks via --task lambada_openai_mt_*.

We currently only support one prompt per task, which we strive to make the "standard" as defined by the benchmark's authors. If you would like to study how varying prompts causes changes in the evaluation score, check out the BigScience fork of this repo. We are currently working on upstreaming this capability to main.

Cluster Usage

The evaluation suite can be called via the Python API, which makes it possible to script jobs with submitit, for example. You can find a detailed example of how this works in scripts/run_eval.py.

Running a job via submitit has two steps: preparing the executor, which controls cluster options, and preparing the actual evaluation options.

First you need to configure the executor. This controls cluster job details, like how many GPUs or nodes to use. For a detailed example, see build_executor in run_eval.py, but a minimal example looks like this:

base_args = {... cluster args ...}
executor = submitit.AutoExecutor(folder="./logs")
executor.update_parameters(**base_args)

Once the executor is prepared, you need to actually run the evaluation task. A detailed example of wrapping the API to make this easy is in the eval_task function, which mainly just calls out to main in scripts/main_eval.py. The basic structure is like this:

def my_task():
    args = {... eval args ...}

    # this is the function from main_eval.py
    main_eval(args, output_path="./hoge.json")

job = executor.submit(my_task)

You can then get output from the job and check that it completed successfully. See run_job for an example of how that works.

Implementing new tasks

To implement a new task in the eval harness, see this guide.

Task Versioning

To help improve reproducibility, all tasks have a VERSION field. When run from the command line, this is reported in a column in the table, or in the "version" field in the evaluator return dict. The purpose of the version is so that if the task definition changes (i.e to fix a bug), then we can know exactly which metrics were computed using the old buggy implementation to avoid unfair comparisons. To enforce this, there are unit tests that make sure the behavior of all tests remains the same as when they were first implemented. Task versions start at 0, and each time a breaking change is made, the version is incremented by one.

When reporting eval harness results, please also report the version of each task. This can be done either with a separate column in the table, or by reporting the task name with the version appended as such: taskname-v0.

Test Set Decontamination

To address concerns about train / test contamination, we provide utilities for comparing results on a benchmark using only the data points nto found in the model training set. Unfortunately, outside of models trained on the Pile and C4, its very rare that people who train models disclose the contents of the training data. However this utility can be useful to evaluate models you have trained on private data, provided you are willing to pre-compute the necessary indices. We provide computed indices for 13-gram exact match deduplication against the Pile, and plan to add additional precomputed dataset indices in the future (including C4 and min-hash LSH deduplication).

For details on text decontamination, see the decontamination guide.

Note that the directory provided to the --decontamination_ngrams_path argument should contain the ngram files and info.json. See the above guide for ngram generation for the pile, this could be adapted for other training sets.

python main.py \
    --model gpt2 \
    --tasks sciq \
    --decontamination_ngrams_path path/containing/training/set/ngrams \
    --device cuda:0

Cite as

@software{eval-harness,
  author       = {Gao, Leo and
                  Tow, Jonathan and
                  Biderman, Stella and
                  Black, Sid and
                  DiPofi, Anthony and
                  Foster, Charles and
                  Golding, Laurence and
                  Hsu, Jeffrey and
                  McDonell, Kyle and
                  Muennighoff, Niklas and
                  Phang, Jason and
                  Reynolds, Laria and
                  Tang, Eric and
                  Thite, Anish and
                  Wang, Ben and
                  Wang, Kevin and
                  Zou, Andy},
  title        = {A framework for few-shot language model evaluation},
  month        = sep,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {v0.0.1},
  doi          = {10.5281/zenodo.5371628},
  url          = {https://doi.org/10.5281/zenodo.5371628}
}