From cead23d03e0488b738db1d03849c3161cd48a1b0 Mon Sep 17 00:00:00 2001 From: Alex Gu Date: Wed, 3 Jan 2024 15:47:44 -0500 Subject: [PATCH] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 57b8976..7df2f7c 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ We provide a script compatible with SLURM to run inference on CRUXEval with Hugg This script parallelizes the 800 samples of the benchmark in a data-parallel fashion across the GPU's. After running the scripts, the generations will appear in `inference/model_generations_raw/shard_i.json`, where `i` ranges from `0` to `GPUS-1`. To convert these into a form that is readily available for evaluation, run `python combine_generations.py`, which will create a file `../model_generations/{MODEL_INFO}/generations.json`. The generations can then be evaluated by following the above instructions. -For best results, we recommend running WizardCoder with `transformers==4.31.0/vllm==0.1.4` and all other models with `transformers==4.34.1/vllm==0.2.2`. WizardCoder performance has been known to degrade with newer versions of transformers. +For best results, we recommend running WizardCoder with `transformers==4.31.0/vllm==0.1.4` and all other models with `transformers==4.36.2/vllm==0.2.6`. WizardCoder performance has been known to degrade with newer versions of transformers. ## 🤖 Running Inference on OpenAI Models You need to use your own API key and comply with OpenAI terms of use. We provide a script to run inference on OpenAI models if you would like to try different temperatures or latest models. Set the `OPENAI_API_KEY` environmental variable to be your API key, for example via `export OPENAI_API_KEY = YOUR_KEY`. Then, `cd openai` and run `python openai_run.py`. Like before, the generations will appear in `../model_generations/{MODEL_INFO}/generations.json`.