[Feature]: Implement and test speculative editing

[neubig]

What problem or use case are you trying to solve?

File editing is not perfect with our current method using SWE-Agent style actions.

Do you have thoughts on the technical implementation?

Cursor has recently published work on speculative editing, that we could possibly incorporate into our agent workflows: https://cursor.sh/blog/instant-apply

We would want to implement a prototype and benchmark it against SWE-Bench.

This would be a good issue for new contributors who have significant experience with LLMs and agents!

[JeffKatzy]

@neubig I'm interested in this, can take a look at this, this weekend.

[JeffKatzy]

@neubig Ok, so I think the easiest approach would be to implement this with huggingface's assistant model feature.

I found some references to this with whisper, and another hf blog here.

So would a valid implementation be to use a hf model, with an assistant model, and then update the do_completion call here, so that it can use the assistant. And then benchmark against using without an assistant?

One downside of using hf is it does not look there's a way to make it compatible with non-hf models like gpt-4, etc. I looked to see if there was a way to implement a hf model wrapper, but didn't see a solution.

[neubig]

Hmm, I'm a bit confused. I think the main difference that we would need to have is that we would need to generate edits in the diff format like that cited on the blog:

@@ ... @@
 function binarySearch(arr, x) {
-    let low = 0, high = arr.length - 1;
-    while (low <= high) {
-        let mid = Math.floor((low + high) / 2);
-        if (arr[mid] === x) {
-            return mid;
-        }
-        low += 1;
-    }
-    return -1;
+    let low = 0, high = arr.length - 1;
+    while (low <= high) {
+        let mid = Math.floor((low + high) / 2);
+        if (arr[mid] === x) {
+            return mid;
+        } else if (arr[mid] < x) {
+            low = mid + 1;
+        } else {
+            high = mid - 1;
+        }
+    }
+    return -1;
 }

I'm not sure how this has something to do with the assistant feature on hugging face. Could you elaborate more?

[JeffKatzy]

Ok, that helps. I was looking at the implementation portion, where they talk about speculative decoding in the blog post.

"We achieve speeds of >1000 tokens/s (just under 4000 char/s) on our 70b model using a speculative-decoding variant tailored for code-edits, called speculative edits."

"Our biggest win comes from our custom speculative decoding algorithm called “speculative edits”. It is equivalent to a full-file-rewrite, while being up to 9x faster."

And the assistant model is how speculative decoding works with huggingface.

Ok, so now I get that you are talking about generating edits in a diff format. I'll look into this.