Skip to content

ryanmahan/copilot-interview

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Can Github Copilot pass a coding interview?

It could get an internship!

Note: the below is out of date by a little, I've run a few more tests but they don't change the results by much so far.

Method

I chose typescript as my language of choice, as providing a function with types would be easier both for me and copilot. I could provide function defintions with types to clarify what I wanted, and copilot could write types into the body so I could read the code easily.

I used leetcode's top 25 most frequently asked questions, consisting of 17 medium, 6 easy, and 1 hard questions. I provided Copilot with a comment describing what I would usually write in a code base to describe a function's purpose. I did not provide the whole prompt to try and avoid copilot from finding exact matches in existing repositories, but sometimes provided the whole prompt as a hail mary. One question, ThreeSum, required the whole description before an answer could be provided.

I tried different prompts multiple times to try different styles of writing comments for copilot. When copilot created a suboptimal solution, I would provide hints at a more optimal one, such as "find the longest common prefix using a trie." In these cases, copilot would generate code for a more optimal solution, but it was not correct.

Results

Copilot was able to answer 13 of the 25 questions. It produced optimal answers for 8 questions. In two cases, it produced somewhat-correct code, where nearly half of the test cases would pass. Only once did it produce code that contained an infinite loop, and once it produced code that would not compile.

What does this mean for copilot? Nothing. It's called copilot for a reason. It certainly made solving questions faster and as a copilot in an interview, it would come in handy.

Specific Results

Check out the files in this repository to see how copilot did on certain questions! All code is generated by copilot, comments are a mix of mine and copilot's. File names generally match the leetcode question, and the CSV contains some tracking and notes I wrote while testing.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published