Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
minimario committed Jan 3, 2024
1 parent 4a94823 commit cead23d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ We provide a script compatible with SLURM to run inference on CRUXEval with Hugg

This script parallelizes the 800 samples of the benchmark in a data-parallel fashion across the GPU's. After running the scripts, the generations will appear in `inference/model_generations_raw/shard_i.json`, where `i` ranges from `0` to `GPUS-1`. To convert these into a form that is readily available for evaluation, run `python combine_generations.py`, which will create a file `../model_generations/{MODEL_INFO}/generations.json`. The generations can then be evaluated by following the above instructions.

For best results, we recommend running WizardCoder with `transformers==4.31.0/vllm==0.1.4` and all other models with `transformers==4.34.1/vllm==0.2.2`. WizardCoder performance has been known to degrade with newer versions of transformers.
For best results, we recommend running WizardCoder with `transformers==4.31.0/vllm==0.1.4` and all other models with `transformers==4.36.2/vllm==0.2.6`. WizardCoder performance has been known to degrade with newer versions of transformers.

## 🤖 Running Inference on OpenAI Models
You need to use your own API key and comply with OpenAI terms of use. We provide a script to run inference on OpenAI models if you would like to try different temperatures or latest models. Set the `OPENAI_API_KEY` environmental variable to be your API key, for example via `export OPENAI_API_KEY = YOUR_KEY`. Then, `cd openai` and run `python openai_run.py`. Like before, the generations will appear in `../model_generations/{MODEL_INFO}/generations.json`.
Expand Down

0 comments on commit cead23d

Please sign in to comment.