Autocomplete: Fix retrieval hints

[philipp-spiess]

I noticed that the context size hints returned values suffixed with chars but one of the values was not going through the same tokensToChars estimation.

This value was used by the context mixer and the retrieval strategies. Luckily, the Jaccard similarity strategy was not using that value. However, the context mixer was likely truncating results because of this, so I would suspect this to increase the average context length a bit (which could have a negative impact on performance). However there was another bug that counter-acted this change a bit. The previous totalFileContextChars was used to only count the additional code snippets (so excluding prefix and suffix) while the documentation said that this number is inclusive.

Also, the actual truncation happens at the prompt level and is unaffected by these hints. These only tell the context retriever how much context it should retrieve.

Some math shows us that the actual difference is not big:

Before:

maxRetrievedChars = maxContextTokens * 0.9

After:

maxRetrievedChars = 4 *  maxContextTokens * 0.9 - 4 * maxContextTokens * 0.6 - 4 * maxContextTokens * 0.1
maxRetrievedChars = 4 * maxContextTokens * ( 0.9 - 0.6 - 0.1 )
maxRetrievedChars = 4 * maxContextTokens * ( 0.2 )
maxRetrievedChars = maxContextTokens * 0.8

So yeah, almost the same 😅

However, with the new change, if fewer actual suffix or prefix characters are used, we will retrieve more context to fill up the window (which IMO is a good thing)

Test plan

• Enable the feature flag
• Make some autocomplete request
• Observe that context snippers are still added.

[philipp-spiess]

Somehow this doesn't add up to 100% haha, let me run some more tests to make sure this still works as expected

[philipp-spiess]

Updated the PR descriptions with my findings. This was a fun one!

[valerybugakov]

Nice find!