Cody: Add a cache for inline completions

[philipp-spiess]

This adds a simple LRU cache for completions with the main goal to further reduce the number of requests and reduce annoying churn in the scenario where a user receives a completion and types the exact same characters. Previously, the new characters would cause a new completion request which could yield different result.

Here's an example of how this cache works:

• Imagine the current file prefix looks like this with the cursor at the end:

  const files = ["a", "b"];
  for(let i = 0;

• We receive the following completion:
  • i < files.length; i++) {
• We now generate create different versions (up until the next \n) of the input prefix by concatenating characters from the completion (Only using the last line here to visualize):
  1. for(let i = 0;
  2. for(let i = 0;
  3. for(let i = 0; i
  4. for(let i = 0; i
  5. for(let i = 0; i <
  6. for(let i = 0; i <
  7. for(let i = 0; i < f
  8. for(let i = 0; i < fi
  9. for(let i = 0; i < fil
  10. for(let i = 0; i < file
  11. ...

Additional thoughts

• I have't added a cache to multiline providers, since these are triggered less often than inline suggestions anyways.
• The LRU cache is limited to 500 file prefixes, regardless of how large these are. We might want to tweak this later. It also currently retain the prompt as part of the Completion interface which may not be necessary.
• I've re-enabled the request that forces adds a \n to the prefix. These can now be reused if you type enter and will result in a faster suggestion for the next line.

Test plan

I've added a console.log when a cache hit is encountered to visualize it while playing around with it:

Screen.Recording.2023-04-24.at.17.54.08.mov

[philipp-spiess]

The limit of 10 characters is arbitrary. We could make this dynamic as well and let it cache for all combinations up to the next \n

I think I'll need to do this right away, 10 characters is just too weirdly arbitrary after some more testing (especially since VS Code doesn't even seem to return completions while typing mid-word, so 10 character barely manages to show up a second time (after ~1 word)

[beyang]

Suggested change

console.log('CACHE HIT')

[philipp-spiess]

😐

[vdavid]

🔪👨🏻‍🍳😄 (killed the waiter—only found cook emoji)