Low Pass@k

[Naman-ntc]

Hi,
Thanks for the great work!
Firstly I wanted to ask about the performance of the code-clippy models. It seems that the 125M parameter models are quite weak and perform quite poorly on human-eval dataset (even lower than GPT-Neo-1.3B?). Any idea why this is happening.

Also is there some update on the evaluation of the GPT-Neo-1.3 B code-clippy model?

Finally, I would love to contribute to upcoming iterations of code-clippy. Should I join the discord channel?

[reshinthadithyan]

Hello, can you exactly point out which model you've been using?
Please read #61 (comment) for more information on the right model to be used. If anything please feel free to mention it here.
Thanks for your interest to contribute. Please join the discord server, we'll post the onboarding document soon.
Thanks, good day!

[Naman-ntc]

Hi,
Sorry, I was just looking at the models wiki page where evaluation on Human-Eval dataset is provided :

Why is pass@k is 0 for all code-clippy models?

[ncoop57]

We aren't exactly sure to be honest. I initial thought is that our training script was not optimized (the model would learn a little and then immediately stop learning). We are actively working to improve our model training both by scaling up models as well as cleaning our data more. Closing this issue for now. If you'd like to help us improve, we are going to be creating a CONTRIBUTION.md guide for new contributors so that we can onboard new volunteers easier. Join our discord if you want to stay up to date with that news!