Autocomplete: Add feature flags to disable network cache and request recycling

[philipp-spiess]

This adds two feature flags to control the request manager behavior:

• cody-autocomplete-disable-network-cache: When set, it won't cache completion requests anymore. My theory here is that this optimization is no longer necessary and when the first line of cache (the last candidate logic) does not hit, we actually want a new completion to be generated. This also makes it so that manual completions (which bypass the lastCandidate check) will now cycle through different completions instead of resolving with the same one. Note that this can still return the same completion from the backend in which case our ID deduplication logic will still work.

• cody-autocomplete-disable-recycling-of-previous-requests: When set, it will not try to complete inflight requests base on the result of previous network requests. Instead, it will abort requests when they are no longer relevent. This was initially added as a mechanism to save latency but there are a few flaws here:

  1. The context for the previous request might be incorrect: E.g. a single-line request is used to respond to a multi-line reuqest.
  2. This prevents requests from being aborted which increases costs
  3. This adds quiete a bit of complexity
  4. The actual impact of this change was never measured as an A/B test

  My theory is that this is still relevant to improve UX and should probably receive fixes for i), however, I want to test if this hypothesis is true and it does indeed meaningfully improve the UX (CAR, Latency)

Test plan

Using the keyboard shortcut to trigger a completion will return one from cache:

With cody-autocomplete-disable-network-cache on, it will do a request:

Writing function bubbleSort( slowly will create a recycled result:

With cody-autocomplete-disable-recycling-of-previous-requests on, the request will take longer and will not be recycled

[abeatrix]

thanks 🙇‍♀️

[abeatrix]

LGTM

[valerybugakov]

This also makes it so that manual completions (which bypass the lastCandidate check) will now cycle through different completions instead of resolving with the same one.

Even if we decide the second cache layer is helpful, let's implement the above behavior 🙏
I want this badly!

[philipp-spiess]

@valerybugakov Yeah, I agree!