Run evaluation on full SWE-Bench

[rezzie-rich]

Love the progress so far!

Will you guys test and publish the full swe-bench and the 25% subset test besides just the swe-bench lite?

On auto-code-rover repo, it says 22% on swe-bench lite and 16% on full swe-bench. However, you guys have ACR at 16% on swe-bench lite. Is that the result you guys got or a typo?

[rbren]

Thanks for pointing that out! Not sure if it was a typo, or if we were using an old result of theirs.

Let's remove the graph until we can generate a better one

[neubig]

This number is from the most recent version of the AutoCodeRover paper! I think we should clarify this in the graph.

[frankxu2004]

From the ACR paper, it shows:

Note that ACR-avg is the comparable number here, as it's the average of 3 runs (meaning that it's pass@1 rate). I see that the 22.33% number inside the repo is the ACR-all, which is the union of 3 runs ( meaning that it's pass@3 rate). I think it's still valid number comparison that for pass@1, ACR is 16%

[rbren]

Ahh thank you for the clarification! I will remove my PR

[neubig]

So I think the AutoCodeRover is fine as-is, but I agree we should still run on all of SWE-Bench. The main bottleneck for this is time and cost, it costs about $6,000 to run on all of SWE-Bench with GPT-4

[libowen2121]

@rezzie-rich Thank you for the question! As @frankxu2004 clarified, we only report the pass@1 results in the graph. Our evaluation containerization only supports SWE-bench-lite for now and we will extend it to support the full test set!