Review limit error metrics & error codes more generally

[mattheworiordan]

See https://support.ably.io/solution/articles/3000082364-explanation-all-metrics-within-rate-limit-errors. I've created this in response to a customer hitting a max channel rate limit, see https://support.ably.io/solution/articles/3000075165-is-there-a-maximum-number-of-channels-per-connection-.

@paddybyers @SimonWoolf can one of you please:

• Confirm that the metrics I have listed are correct. I had to guess the metric IDs.
• Please review https://support.ably.io/solution/folders/3000012310 to see if we have articles for the error codes relating to common rate limit issues. If there are any missing, please post them here and we can ask @tomczoink to help.

┆Issue is synchronized with this Jira Task by Unito

[SimonWoolf]

Confirm that the metrics I have listed are correct

• channels.maxRate should be channel.maxRate (as within a single channel)

• For acct-wide limits, currently the message you get when you try to publish doesn't include a metric, it just says eg Maximum account-wide instantaneous messages rate exceeded, that seemed friendlier. But I can easily make it Maximum account-wide instantaneous rate limit exceeded; metric = messages.maxRate or something, if you'd prefer people to have a metric they can look up.

• reactor limits are missing: 'reactor.httpEvent.maxRate', 'reactor.amqp.maxRate' (assuming I make the change above)

• For example, you have hit an instantaneous per second rate limit, then we will only reject new published messages for that second, and the block will be removed in the following second interval -- this isn't really correct. For things like per-channel and per-connection limits, the block interval is 6 seconds long. For account-wide rate limits there isn't a block interval, instead suppression is done on a rolling probabilistic basis (see 'instantaneous limits' section of https://support.ably.io/support/solutions/articles/3000079684-understanding-account-limit-notifications-within-email-alerts-or-in-your-dashboard )

[mattheworiordan]

Thanks @SimonWoolf

Yeh, I was being a bit lazy with my per second rate limit description, also trying to keep it simple to understand, but it's wrong so I will update and be less lazy

[mattheworiordan]

@SimonWoolf re: my request for common error codes, just got this from a customer 30 minutes ago:

Thanks for contacting, we’ve took a look at the current situation. https://www.ably.io/accounts/1391/notifications states we’ve hit the apiRequests limit, not the messages limit. Also, the error code given is not documented: https://help.ably.io/error/40115

Would it be possible to please get that list of the most common error codes so that we can create appropriate solution articles?

[SimonWoolf]

Would it be possible to please get that list of the most common error codes so that we can create appropriate solution articles?

what list of most common error codes?

[mattheworiordan]

See task above in this issue:

Please review https://support.ably.io/solution/folders/3000012310 to see if we have articles for the error codes relating to common rate limit issues. If there are any missing, please post them here and we can ask @tomczoink to help.

[SimonWoolf]

...So now I'm looking at this, they're not very consistent. Per-connection publish rate limit is 42911 (or 42921 if fatal), and acct-wide rate limits are also 42911, but most other rate limits (e.g. per-channel) are 42910.

ISTM we should either just use the same code for all rate limits (well, two, one fatal and one nonfatal), or have a different code for every different rate limit. WDYT? @mattheworiordan @paddybyers

[paddybyers]

It depends on whether or not we want to be able to link to different help docs in those cases. Or perhaps use a limited number of codes, and exploit the functionality of being able to construct an href base on information beyond just the code.

If we do go for a different code in each case, then it should be a whole new family starting at 10000 or something.

[mattheworiordan]

Well I think it's better to have unique error codes for logical grouping of limits so that we can write simple articles to address that problem. For example, how to address message.maxRate problems (fan-out) is very different to address tokenRequest hourly limits. Equally, having different codes for all limits could be hard.

Please can you suggest some natural groupings and we'll get @tomczoink to write up articles we can build on.

[paddybyers]

OK, so lets have completely new series of codes that get sent along with a 429 statusCode for different rate limits.

10000-11999 - rate limit errors
10000-10999 - hard limit errors - ie the attempted operation was rejected or modified as a result of the rate limit
11000-11999 - warning limit codes - the attempted operation was permitted, but the current usage is close to hitting a hard limit.

The individual codes will be consistent in the 10xxx and 11xxx series.

Then lets break it down by functional area:

100xx: generic/unspecified
1001x: instantaneous limit
1002x: hourly/monthly limit
1003x: payload/request size limit

101xx: message limits
1011x: instantaneous limi
1022x: hourly/monthly limit
1023x: size limit

102xx: connection limits
1021x: instantaneous limit
1023x: other limit (eg max attachments)

103xx: channel limits
1031x: instantaneous limit
1033x: other limit (eg max presence)

104xx: request (api request/ token request) limits
1041x: instantaneous limit
1042x: hourly/monthly limit
1043x: size limit

105xx: reactor limits
1051x: instantaneous limit
1052x: hourly/monthly limit
1053x: size limit

[mattheworiordan]

I like this. Would it be possible to agree on keeping the error codes in one place (a bit like what we did with ably/ably-ruby@7df0916) so that once implemented, we could ask @tomczoink to create pages for each error code? I realise it's a lot of support articles, but they will be quite generic with slight variations and links to larger articles.

[paddybyers]

Would it be possible to agree on keeping the error codes in one place

You mean like in ably-common as now, or something different?

[mattheworiordan]

Would it be possible to agree on keeping the error codes in one place (a bit like what we did with ably/ably-ruby@7df0916)

Well yes. I was thinking about how realtime handles it, but of course that's not really all that important if we update ably-common. So ignore me if it's going to go into ably-common.