Autocomplete: Expose rate limiting and other errors to users

[toolmantim]

There are some rate limits in place, both for chats and autocomplete. For chat, we display error messages in the transcript, so users know that their chat message failed because of rate limit exhaustion. But for autocomplete, we give no visible feedback.

We need to:

• Make it visible there's a rate limit happening
• Give them enough information to understand what's happening
• Know what they can do about it (wait, read the docs?)

From @eseliger’s Slack message:

For autocompletions, as far as I know we don’t indicate errors too heavily today.
We’ve been thinking of making the Cody navbar item turn red or something like that and on hover show it’s not available at the moment due to rate limits.

• Design
• Reviewed & ready to go

[toolmantim]

@eseliger @philipp-spiess I've done a bit of testing locally with adding an additional icon, or seeing if it's possible to do the split/two-button style (it doesn't appear to be possible), and landed exactly back at what you suggested:

(This is from a slightly new design than what's on main, so please ignore the slightly different statusbar label and quickpick items)

Use the current icon, change the background and tooltip, and add an item to the top of the quickpick (selected, with a separator below).

@eseliger How does the rate limiting logic work? And what error message/info do we have available to show? An example of whatever string the backend spits out that we'd show to the user could do, just so I can sanity check it with reality.

As far as docs go, I found https://docs.sourcegraph.com/cody/explanations/cody_gateway#rate-limits-and-quotas — should we update it while we're here?

[eseliger]

@eseliger How does the rate limiting logic work? And what error message/info do we have available to show? An example of whatever string the backend spits out that we'd show to the user could do, just so I can sanity check it with reality.

You will get 429 HTTP errors when making requests while throttled, with headers that tells when the limit resets and so forth:

	w.Header().Set("x-ratelimit-limit", strconv.Itoa(err.Limit))
	w.Header().Set("x-ratelimit-remaining", strconv.Itoa(max(err.Limit-err.Used, 0)))
	w.Header().Set("retry-after", err.RetryAfter.Format(time.RFC1123))

They can do:

• For app, reach out to us on discord
• For enterprise, talk to their sales rep
• Or simply, wait

[toolmantim]

Thanks! In that case, I think the UI should probably just say "Retry again in {friendlyTimeDuration}" type thing (e.g. "Retry again in 2 minutes") — but in the documentation we can have that information for how to request a limit increase.

What are the time buckets? 1 minute?

[toolmantim]

For the doc update:

For app, reach out to us on discord

Is there a specific channel/person/group they should talk to?

Does that also apply to sourcegraph.com users?

[philipp-spiess]

Thanks! In that case, I think the UI should probably just say "Retry again in {friendlyTimeDuration}" type thing (e.g. "Retry again in 2 minutes") — but in the documentation we can have that information for how to request a limit increase.

What are the time buckets? 1 minute?

The buckets are 24 hours. And I think for autocomplete, retry is not really a useful term since the code change was likely already made. For that feature, we should say something like "Autocomplete will be available again in {friendlyTimeDurationi}. {Some CTA} now to increase your quota" or so.

[toolmantim]

Thanks @philipp-spiess, will update.

(Is anyone hitting this on their first day of use? That's a long time to wait…)

[eseliger]

(Is anyone hitting this on their first day of use? That's a long time to wait…)

Note that this is instance-wide, so for enterprise customers with 500 licensed users, all 500 combined have to on average use the per-user allocated number of requests.

Is there a specific channel/person/group they should talk to?

Not sure, @jdorfman usually handles these I think.

Does that also apply to sourcegraph.com users?

Yes, dotcom and app :)

[toolmantim]

Thanks @eseliger, will get a design & docs update ready.

It might be a nice improvement to reduce the bucket duration to 30min at some point… 24h seems a bit unfriendly (it's more like a 1 day ban than a rate limit).

[eseliger]

It might be a nice improvement to reduce the bucket duration to 30min at some point… 24h seems a bit unfriendly (it's more like a 1 day ban than a rate limit).

While our rate limits depend on requests made, not spend incurred on our end or so, and because we don't have rollover in additional windows, using a 1h window might actually make it worse for most of our customers, because:

• To keep cost in a reasonable frame, we define a per user per month spend for these features
• Since we don't track tokens due to API limitations, we estimated what the average request will cost, so our gating metric is reqs
• Then we distribute that amount evenly over the month, currently by day

For an instance of 5000 users that can do 1000 requests per user per day, that means: All users combined can make 5M requests in a 24h window
If we switch to a 1h window, they will get 208k requests an hour, again, no rollover.

Now, when the work day starts and everyone's hacking away happily, they might want to burn through their 24h limit in the 8h of their workday. With the 24h window, they get all their 5M permitted requests during the work day. If it was an hourly window, they would only get to use 1/3 of that during their working hours.

There's more changes coming to billing here likely as we get closer to GA, do you think we need to revisit this before then? Ideally, 95%+ of users should not hit their limits, at least not enterprise users, those numbers were chosen carefully that most people get through the day without interruptions, favoring user experience, while keeping cost incurred fair.

[toolmantim]

Thanks for getting me up to speed. The Enterprise case makes sense as is. I think we'd only need to revisit if we see that free users are hitting it during their first days of use, and getting stuck for a whole day.

[toolmantim]

Here's an updated design, based on those words @philipp-spiess, and I reckon this is good to go:

Give it's a single line, I've left off the CTA in the text, and assume they'll discover to click the item and read the docs page.

[toolmantim]

(I don't love "notice" either — suggestions welcome!)

[jdorfman]

Is there a specific channel/person/group they should talk to?

Users usually just drop a message in #help, and me or a community member like @deepak2431 flags it and I'll takes care of the request.

[philipp-spiess]

I think we should probably work on this sooner rather than later. Dotcom users are getting hit by this and are confused: https://github.com/sourcegraph/cody/discussions/660

@toolmantim Is this ready to be taken over from your side? 🙂

[toolmantim]

Yeah this is good to go @philipp-spiess!