Track number of hosts affected by the rate limiter #13541

MadLittleMods · 2022-08-16T18:34:39Z

Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation.

Follow-up to #13534

Part of #13356

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

Related to #13499 Mentioned in https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.zjko88lr25j

See #13534 (comment)

Follow-up to #13534 Part of #13356

synapse/util/ratelimitutils.py

MadLittleMods · 2022-08-16T18:48:46Z

changelog.d/13541.misc

@@ -0,0 +1 @@
+Add metrics to track how the rate limiter is affecting requests (sleep/reject).


Same changelog as #13534 so they merge together.

Related to: - #13534 - #13541

…iter

…littlemods/metrics-rate-limit-affected-host-count Conflicts: synapse/util/ratelimitutils.py

synapse/util/ratelimitutils.py

reivilibre

Seems reasonable; minor suggestions

synapse/util/ratelimitutils.py

See: - #13541 (comment) - #13541 (comment)

…-host-count Conflicts: synapse/util/ratelimitutils.py

MadLittleMods · 2022-08-18T15:05:30Z

Thanks for the review @erikjohnston and @reivilibre 🐘

MadLittleMods · 2022-08-22T21:24:32Z

synapse/util/ratelimitutils.py

+            lambda: sum(
+                ratelimiter.should_reject()
+                for ratelimiter in self.ratelimiters.values()
+            ),


This seems to always evalulate to 0.

Prometheus data: https://prometheus.matrix.org/graph?g0.expr=synapse_rate_limit_reject_affected_hosts&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h&g0.end_input=2022-08-22%2021%3A21%3A52&g0.moment_input=2022-08-22%2021%3A21%3A52

Even though we see individual requests being rejected:

As a note, when I log should_reject() in Complement, I see should_reject queue_size=0 self.reject_limit=99999 (the high reject_limit is from docker/complement/conf/workers-shared-extra.yaml.j2#L36) for all of the calls. Not sure I am exercising this enough in parallel to see it though.

This is probably caused by #13641 which is being addressed in #13649

I've tried to visualise this at https://grafana.matrix.org/d/000000012/synapse?orgId=1&viewPanel=225&var-datasource=default&var-bucket_size=$__auto_interval_bucket_size&var-instance=matrix.org&var-job=All&var-index=All but I still see 0 for all rejected metrics, even with #13649.

These numbers look consistent with data I saw when investigating in the manhole (sum(x.should_reject() for x in hs.get_federation_ratelimiter().ratelimiters.values())). It might be that we need to wait some time for queues to grow before we start rejecting requests. I'd like to leave confirming/investigating this on your plate @MadLittleMods if that's ok?

@DMRobertson Looks like #13649 hasn't fixed it if it has already been rolled out to matrix.org 😩 (still all 0)

Here are the rate limiter graph section I have:
https://grafana.matrix.org/d/dYoRgTgVz/messages-timing?orgId=1&from=1661854473240&to=1661876073240&viewPanel=220

I'll create an actual issue to track this more long-term now ⏩

Synapse 1.66.0rc1 (2022-08-23) ============================== This release removes the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. This removal was originally planned for Synapse 1.64, but was later deferred until now. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Features -------- - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](#13188), [\#13563](#13563)) - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](#13503)) - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](matrix-org/matrix-spec-proposals#3852). ([\#13549](#13549)) - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](matrix-org/matrix-spec-proposals#2716). ([\#13551](#13551)) - Add support for compression to federation responses. ([\#13537](#13537)) - Improve performance of sending messages in rooms with thousands of local users. ([\#13522](#13522), [\#13547](#13547)) Bugfixes -------- - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](#13514)) - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](#13525)) - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](#13566)) - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](#13574)) Improved Documentation ---------------------- - Add `openssl` example for generating registration HMAC digest. ([\#13472](#13472)) - Tidy up Synapse's README. ([\#13491](#13491)) - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](#13492)) - Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](#13497)) - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](#13515)) - Add missing links in `user_consent` section of configuration manual. ([\#13536](#13536)) - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](#13538)) Deprecations and Removals ------------------------- - Remove the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Internal Changes ---------------- - Update the rejected state of events during de-partial-stating. ([\#13459](#13459)) - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](#13477)) - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](#13531)) - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](#13453)) - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](#13489)) - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](#13499), [\#13554](#13554)) - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](#13533)) - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](#13534), [\#13541](#13541)) - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](#13535), [\#13584](#13584)) - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](#13544)) - Update metrics to track `/messages` response time by room size. ([\#13545](#13545)) - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](#13024)) - Clean-up tests for notifications. ([\#13471](#13471)) - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](#13474)) - Use literals in place of `HTTPStatus` constants in tests. ([\#13479](#13479), [\#13488](#13488)) - Add comments about how event push actions are rotated. ([\#13485](#13485)) - Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](#13493)) - Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](#13502)) - Reduce the number of tests using legacy TCP replication. ([\#13543](#13543)) - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](#13549)) - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](#13558))

Synapse 1.66.0 (2022-08-31) =========================== No significant changes since 1.66.0rc2. This release removes the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. This removal was originally planned for Synapse 1.64, but was later deferred until now. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Deployments with multiple workers should note that the direct TCP replication configuration was deprecated in Synapse v1.18.0 and will be removed in Synapse v1.67.0. In particular, the TCP `replication` [listener](https://matrix-org.github.io/synapse/v1.66/usage/configuration/config_documentation.html#listeners) type (not to be confused with the `replication` resource on the `http` listener type) and the `worker_replication_port` config option will be removed . To migrate to Redis, add the [`redis` config](https://matrix-org.github.io/synapse/v1.66/workers.html#shared-configuration), then remove the TCP `replication` listener from config of the master and `worker_replication_port` from worker config. Note that a HTTP listener with a `replication` resource is still required. See the [worker documentation](https://matrix-org.github.io/synapse/v1.66/workers.html) for more details. Synapse 1.66.0rc2 (2022-08-30) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.66.0rc1 where the new rate limit metrics were misreported (`synapse_rate_limit_sleep_affected_hosts`, `synapse_rate_limit_reject_affected_hosts`). ([\matrix-org#13649](matrix-org#13649)) Synapse 1.66.0rc1 (2022-08-23) ============================== Features -------- - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\matrix-org#13188](matrix-org#13188), [\matrix-org#13563](matrix-org#13563)) - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\matrix-org#13503](matrix-org#13503)) - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](matrix-org/matrix-spec-proposals#3852). ([\matrix-org#13549](matrix-org#13549)) - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#13551](matrix-org#13551)) - Add support for compression to federation responses. ([\matrix-org#13537](matrix-org#13537)) - Improve performance of sending messages in rooms with thousands of local users. ([\matrix-org#13522](matrix-org#13522), [\matrix-org#13547](matrix-org#13547)) Bugfixes -------- - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\matrix-org#13514](matrix-org#13514)) - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\matrix-org#13525](matrix-org#13525)) - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\matrix-org#13566](matrix-org#13566)) - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\matrix-org#13574](matrix-org#13574)) Improved Documentation ---------------------- - Add `openssl` example for generating registration HMAC digest. ([\matrix-org#13472](matrix-org#13472)) - Tidy up Synapse's README. ([\matrix-org#13491](matrix-org#13491)) - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\matrix-org#13492](matrix-org#13492)) - Add a warning to retention documentation regarding the possibility of database corruption. ([\matrix-org#13497](matrix-org#13497)) - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\matrix-org#13515](matrix-org#13515)) - Add missing links in `user_consent` section of configuration manual. ([\matrix-org#13536](matrix-org#13536)) - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\matrix-org#13538](matrix-org#13538)) Deprecations and Removals ------------------------- - Remove the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Internal Changes ---------------- - Update the rejected state of events during de-partial-stating. ([\matrix-org#13459](matrix-org#13459)) - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\matrix-org#13477](matrix-org#13477)) - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\matrix-org#13531](matrix-org#13531)) - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\matrix-org#13453](matrix-org#13453)) - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\matrix-org#13489](matrix-org#13489)) - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\matrix-org#13499](matrix-org#13499), [\matrix-org#13554](matrix-org#13554)) - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\matrix-org#13533](matrix-org#13533)) - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\matrix-org#13534](matrix-org#13534), [\matrix-org#13541](matrix-org#13541)) - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\matrix-org#13535](matrix-org#13535), [\matrix-org#13584](matrix-org#13584)) - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\matrix-org#13544](matrix-org#13544)) - Update metrics to track `/messages` response time by room size. ([\matrix-org#13545](matrix-org#13545)) - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\matrix-org#13024](matrix-org#13024)) - Clean-up tests for notifications. ([\matrix-org#13471](matrix-org#13471)) - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\matrix-org#13474](matrix-org#13474)) - Use literals in place of `HTTPStatus` constants in tests. ([\matrix-org#13479](matrix-org#13479), [\matrix-org#13488](matrix-org#13488)) - Add comments about how event push actions are rotated. ([\matrix-org#13485](matrix-org#13485)) - Modify HTML template content to better support mobile devices' screen sizes. ([\matrix-org#13493](matrix-org#13493)) - Add a linter script which will reject non-strict types in Pydantic models. ([\matrix-org#13502](matrix-org#13502)) - Reduce the number of tests using legacy TCP replication. ([\matrix-org#13543](matrix-org#13543)) - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\matrix-org#13549](matrix-org#13549)) - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\matrix-org#13558](matrix-org#13558)) # -----BEGIN PGP SIGNATURE----- # # iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmMPT8QACgkQ+84Kzgcy # GG9CUAv+Pv/iDpE2jKlV7zQ/cagaKCGsFK5jy0+K9Wr215nP89tuhU37bJXsgvVu # GP3A8k1c/ENPhXwYHLCnnxV3jick1FuVE0W6h0j2PMYeIGNCQhDswytnsQO4JExg # fGLL4ygCzpe8bFX9+mhIM4z8xkZjZX3lIa8CN2LtRLIo0m7qoT1ZWqdt7kAjj5yL # XMk+3Y1yq/Y4SHHqgKurBNdwNcwnv7ynchWxTYa12WVTINt26dLV0Syk3p8u2SLl # 5YNzcDs2TAM7+VxAu7E0AQl426+Ufi122Oj1ZBUG2FxTPLH8Xr18cN2M/at6WxoX # 8pOkGiuahKKvahw1iCoHAGIC66gFIPxBE9xW4R2SKrQtG4sDuKJI0kvunRV8+cy5 # TuJ9cmdDmJR2vj3P3OULqLXGkWsGNJqfZZF8OWkHEI8LUIXZLrAZocFtlonkr9rV # Y8r8LxL8Id1rbHAnCXcJnYdaJ6ol0RIObDFpitY/D8BDUONVw/byeOyAEkq/XPrZ # Ke/9K8sy # =eg1L # -----END PGP SIGNATURE----- # gpg: Signature made Wed Aug 31 13:10:44 2022 BST # gpg: using RSA key 58C4E75BC67C92169A7FDD11FBCE0ACE0732186F # gpg: Can't check signature: No public key # Conflicts: # synapse/api/auth.py # synapse/push/baserules.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/storage/databases/main/event_push_actions.py # tests/server_notices/test_resource_limits_server_notices.py

MadLittleMods added 5 commits August 15, 2022 21:46

Add metrics to track how the rate limiter is affecting requests

325cadc

Related to #13499 Mentioned in https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.zjko88lr25j

Add changelog

3267318

Fix lints

5679bb2

Remove unbounded host from labels

149ac1d

See #13534 (comment)

Track number of hosts affected by the rate limiter

8be321f

Follow-up to #13534 Part of #13356

MadLittleMods added T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. A-Metrics metrics, measures, stuff we put in Prometheus labels Aug 16, 2022

MadLittleMods commented Aug 16, 2022

View reviewed changes

synapse/util/ratelimitutils.py Show resolved Hide resolved

MadLittleMods commented Aug 16, 2022

View reviewed changes

synapse/util/ratelimitutils.py Show resolved Hide resolved

Add changelog and fix lints

de1bbb8

MadLittleMods commented Aug 16, 2022

View reviewed changes

MadLittleMods mentioned this pull request Aug 16, 2022

Add metrics to track how the rate limiter is affecting requests (sleep/reject) #13534

Merged

4 tasks

MadLittleMods added a commit that referenced this pull request Aug 16, 2022

Add metrics to track rate limiter queue timing

e6c2b24

Related to: - #13534 - #13541

MadLittleMods mentioned this pull request Aug 16, 2022

Add metrics to track rate limiter queue timing #13544

Merged

4 tasks

MadLittleMods added 2 commits August 16, 2022 16:38

Merge branch 'develop' into madlittlemods/track-metrics-from-rate-lim…

75ca101

…iter

Merge branch 'madlittlemods/track-metrics-from-rate-limiter' into mad…

5c7d804

…littlemods/metrics-rate-limit-affected-host-count Conflicts: synapse/util/ratelimitutils.py

MadLittleMods marked this pull request as ready for review August 16, 2022 22:09

MadLittleMods requested a review from a team as a code owner August 16, 2022 22:09

erikjohnston reviewed Aug 17, 2022

View reviewed changes

synapse/util/ratelimitutils.py Show resolved Hide resolved

synapse/util/ratelimitutils.py Show resolved Hide resolved

synapse/util/ratelimitutils.py Outdated Show resolved Hide resolved

reivilibre suggested changes Aug 17, 2022

View reviewed changes

synapse/util/ratelimitutils.py Show resolved Hide resolved

synapse/util/ratelimitutils.py Show resolved Hide resolved

synapse/util/ratelimitutils.py Outdated Show resolved Hide resolved

Use simple sum solution

7dc6869

See: - #13541 (comment) - #13541 (comment)

Base automatically changed from madlittlemods/track-metrics-from-rate-limiter to develop August 17, 2022 21:10

Merge branch 'develop' into madlittlemods/metrics-rate-limit-affected…

b9d4be3

…-host-count Conflicts: synapse/util/ratelimitutils.py

MadLittleMods requested review from reivilibre and erikjohnston August 17, 2022 22:08

reivilibre approved these changes Aug 18, 2022

View reviewed changes

MadLittleMods merged commit d64653d into develop Aug 18, 2022

MadLittleMods deleted the madlittlemods/metrics-rate-limit-affected-host-count branch August 18, 2022 15:05

MadLittleMods commented Aug 22, 2022

View reviewed changes

This was referenced Aug 22, 2022

Instrument _check_sigs_and_hash_and_fetch to trace time spent in child concurrent calls #13588

Merged

conf/workers-shared-extra.yaml.j2 is applied to monolith Complement runs #13590

Open

This was referenced Aug 26, 2022

synapse_rate_limit_sleep_affected_hosts registered twice #13641

Closed

Fix rate limit metrics registering twice and misreporting #13649

Merged

Make sure synapse_rate_limit_reject_affected_hosts does what it says it does #13670

Closed

MadLittleMods mentioned this pull request Oct 28, 2022

Reconcile duplication between "Hosts being rate limited" panel vs the "Federation rate limiter" section in Grafana #14315

Closed

matrixbot mentioned this pull request Dec 21, 2023

conf/workers-shared-extra.yaml.j2 is applied to monolith Complement runs element-hq/synapse#13590

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track number of hosts affected by the rate limiter #13541

Track number of hosts affected by the rate limiter #13541

MadLittleMods commented Aug 16, 2022 •

edited

Loading

MadLittleMods Aug 16, 2022

reivilibre left a comment

MadLittleMods commented Aug 18, 2022

MadLittleMods Aug 22, 2022

MadLittleMods Aug 23, 2022

MadLittleMods Aug 26, 2022

DMRobertson Aug 30, 2022 •

edited

Loading

DMRobertson Aug 30, 2022

MadLittleMods Aug 30, 2022

MadLittleMods Aug 30, 2022

		@@ -0,0 +1 @@
		Add metrics to track how the rate limiter is affecting requests (sleep/reject).

Track number of hosts affected by the rate limiter #13541

Track number of hosts affected by the rate limiter #13541

Conversation

MadLittleMods commented Aug 16, 2022 • edited Loading

Pull Request Checklist

MadLittleMods Aug 16, 2022

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

MadLittleMods commented Aug 18, 2022

MadLittleMods Aug 22, 2022

Choose a reason for hiding this comment

MadLittleMods Aug 23, 2022

Choose a reason for hiding this comment

MadLittleMods Aug 26, 2022

Choose a reason for hiding this comment

DMRobertson Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

DMRobertson Aug 30, 2022

Choose a reason for hiding this comment

MadLittleMods Aug 30, 2022

Choose a reason for hiding this comment

MadLittleMods Aug 30, 2022

Choose a reason for hiding this comment

MadLittleMods commented Aug 16, 2022 •

edited

Loading

DMRobertson Aug 30, 2022 •

edited

Loading