Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'senior' coder test suite #141

Closed
the-crypt-keeper opened this issue Jan 5, 2024 · 1 comment
Closed

'senior' coder test suite #141

the-crypt-keeper opened this issue Jan 5, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@the-crypt-keeper
Copy link
Owner

The junior-v2 interview is showing it's age, I created it back when llama was all we had and at the time every single open source model failed the test.

The clustering we now see at the top of the leaderboard is a result of the massive improvements in open source coding models these past 6 months, anything above .95 is a binary pass and junior-v2 has no comparing ability up here.

A more difficult test suite is needed.
@the-crypt-keeper the-crypt-keeper added the enhancement New feature or request label Jan 5, 2024
the-crypt-keeper pushed a commit that referenced this issue Jan 5, 2024
#141 senior coder test suite wip
15cd57a
the-crypt-keeper pushed a commit that referenced this issue Jan 5, 2024
#141 implement idea of check weights
a5a917f 
improve eval console output and add colors
@the-crypt-keeper
Copy link
Owner Author

A senior interview suite mvp is now available, gpt4 can just barely pass it.

If you have any good ideas for interview questions please open PRs!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@the-crypt-keeper