[P1] Confirmation of alpaca_eval version

[BaohaoLiao]

Hi,

I want to confirm the version of alpaca_eval. In the paper and the README, you said you use v1.0. May I ask how you install it? If I install as your recommendation:
pip install alpaca-eval

It is the latest version. Should I install as:

pip install alpaca-eval==v0.1.0

[frankaging]

@BaohaoLiao hey thanks for your question! the version here is okay. You can install the newest version.

Alpaca-Evalv1.0 means we evaluate against text-davinci-003, not against GPT4. You can follow this to do the evaluation:
https://github.com/stanfordnlp/pyreft/tree/main/examples/loreft#offline-evaluation-with-alpaca-eval-v10

Alpaca-Evalv2.0 uses GPT-4. To be competitive with v2.0, please train with longer sequence length. We are training with max sequence length of 768, this is too limited. We have to do it to keep a fair comparison with previous works.

[frankaging]

I am closing this issue; feel free to reopen if you have more questions! ty!

[BaohaoLiao]

Thank you for the details. Is it possible to release the generated outputs for alpaca_eval? I.e. the output from Llama-2 7B & LoReFT (ours) in Table 3.

It would be great if you could also release the evaluation results from alpaca_eval.

[frankaging]

@BaohaoLiao unfortunately, these results are removed locally from my disk -- but it should be very easy to reproduce. i think you just need to follow the command we give in the README. let me know if you encounter any issues.