Add Aligner-2B+Qwen1.5-72B-Chat & Aligner-2B+Claude3 Opus to AlpacaEval #259
Conversation
|
Woaw @AlignInc those are amazing results, and really cool that you can run your method on any (even closed) model! Unfortunately, I just merged the length-controlled (LC) PR to main and as a result there's a merge conflict. Can you please pull from main and run The good news is that your model should perform even better on LC AlpacaEval. Lmk if you face any issues! |
|
Hi! @YannDubs, |
|
Why adding cohere to requirements? |
| @@ -35,7 +35,7 @@ | |||
| ] | |||
| PACKAGES_ALL_API = [ | |||
| "anthropic>=0.18", | |||
| "cohere", | |||
| "cohere<5.0.0a0", | |||
There was a problem hiding this comment.
We are importing class cohere.CohereError here:
The CohereError was removed in cohere v5, which was released yesterday (release history on PyPI).
cohere.CohereError in cohere v4: https://github.com/cohere-ai/cohere-python/blob/v4/cohere/__init__.py#L5
cohere.*Error in cohere v5: https://github.com/cohere-ai/cohere-python/blob/67620c348329308186d0b7e771a06795ea718226/src/cohere/__init__.py#L122-L130
There was a problem hiding this comment.
That makes sense, thanks!
| @@ -35,7 +35,7 @@ | |||
| ] | |||
| PACKAGES_ALL_API = [ | |||
| "anthropic>=0.18", | |||
| "cohere", | |||
| "cohere<5.0.0a0", | |||
There was a problem hiding this comment.
That makes sense, thanks!
requirements.txt
Outdated
| @@ -1,3 +1,4 @@ | |||
| cohere<5.0.0a0 | |||
There was a problem hiding this comment.
But cohere should not be a main requirement, please remove this line!
it can be a requirement in setup.py if you use [all]
|
Congrats @AlignInc, those a really impressive results and I’m looking forward to see how the community picks it up 💯 sorry for the additional work you had to do for this PR! |
|
Thanks for your time~ |
We would like to add Aligner-2B+Qwen1.5-72B-Chat & Aligner-2B+Claude3 Opus to AlpacaEval 2.0. Thank you for such a valuable leaderboard!
It is the reproduction of the paper - Aligner: Achieving Efficient Alignment through Weak-to-Strong Correction
Arxiv url: https://arxiv.org/pdf/2402.02416.pdf
Core insight: It is more easier to learn the correctional residual difference between bad and good answers than to directly master the generation of good answers.