Autocomplete: Various latency related tweaks and new eager cancellation experiment

[philipp-spiess]

A few small tweaks from my learnings of looking at some traces:

1. Fixes a bug where the debounce time was increased for non-local models
2. Sets the same debounce time for single-line and multi-line
3. Remove some config evaluations off the critical path. Those are heavily cached but it would still cause the very first completion to be slower
4. Add a new eager cancellation experiment that will cancel requests as soon as a new request is created and reduces the debounce time significantly to try and counter the latency regression

Test plan

• For the new experiment, I added a abort handler in the fireworks client and ensured it was heavily hit
• For the rest, just made sure completions still work. The changes are trivial.

[valerybugakov]

🔥 ☄️

[valerybugakov]

Can we keep this functionality when eagerCancellation === true? We can throttle (is it the right word here?) requests — with low debounce, we will have them almost on every keystroke, but instead of canceling all of them but the last one, we can keep on request in every number of requests made in the previous 100ms + always keep the tail one.

This way, we preserve the nice UX where the completion is generated early, and a user continues typing as suggested AND decrease the tail completion delay by 65ms. WDYT?

[philipp-spiess]

I’m not sure I understand this right. If we want to keep this, we would have to keep more than just the last request alive (so that another request can actually answer another one). Are you recommending we keep all but the last 2 completions active?

[valerybugakov]

Discussed on the call. We're going to follow-up on that in a separate PR.