Enable rate limiting on -dev

[AlexandraMoga]

Describe the problem and steps to reproduce it:

Currently add-ons submission and API throttling is enabled only on stage. When an issue that affects this functionality is closed, we usually have to wait until the code lands in stage to be able to verify it. By enabling rate limiting on -dev, we can have such issues tested earlier.

As far as I'm aware, we currently have the following throttles:

• for add-on submissions => can be bypassed by adding a user to the language pack submission group - see check LANGPACK_SUBMIT permission for throttling override rather than setting addons-server#12158
• ratings submission => can be bypassed by assigning the Ratings:BypassThrottling to a user - see Allow users with Ratings:BypassThrottling permission to bypass ratings API throttling addons-server#17673
• abuse report submission? - I remember being a daily throttle for this but I'm not sure if there is a possibility to bypass it; in any case, I don't think we run into that rate limit too often.

I've already raised this issue to @diox on slack and, if everyone else agrees, we can re-enable throttling on -dev.

[diox]

We tried to do that but the new API doesn't have the same bypasses yet so it's a bit annoying. Instead, we want to add a bypass mechanism directly in the throttle classes so that it applies to everything automatically. (And we can probably keep the existing code to bypass throttling on ratings/old signing API for now)

[diox]

I gave the new API:BypassThrottling permission to the 2 groups mentioned, and started using that permission instead of the previously existing ones to bypass rate limiting everywhere on the site. And re-activated rate limiting on dev.

[AlexandraMoga]

I've verified the new permission to successfully bypass the following throttles:

• add-on submissions through Devhub
• addon submission through the new submissions API
• addon submission with the old signing API
• posting multiple ratings
• posting abuse reports

Without the permission, throttling still applies as before.

[ioanarusiczki]

@diox

I tested the abuse reports throttles on -stage a month ago when I was writing a test case. I remembered something and checked it again with -dev:

I can send with Postman using the APIs 20 mixed requests (to report a user or an addon , authenticated with Session ID or anonymous ) then I hit the 429, Too Many requests.

Then if I try it using a browser (same PC) I can send 20 more reports for addons/users.

Should I receive a 429 per IP address after the first 20 attempts ?

[diox]

Yeah it should be 20 reports per IP per day regardless of how they were posted. If you are sure this is using the same IP (no proxy/VPN/etc) then please file an issue.

[AlexandraMoga]

@diox Just for the record, I've tried the following method to test the API throttling for abuse reports (no proxy/VPN on):

1. I've sent 20 authenticated requests using a python requests script until I've hit a user_abuse rate limit (recorded in this activity log)
2. I've switched to the browser, logged in with a different account and tried to submit an addon abuse report: the addons manager abuse report window opened, I've filled in the required fields and submitted the abuse; there was no confirmation message received and when looking in the browser console I could see the following error message: Reporting add-on for abuse via firefox failed: Error: Error submitting abuse report which indicates that the abuse report was not posted; I've checked the admin abuse reports and there was indeed no entry created for this attempt; I've checked the activity logs and I found an ip_abuse entry for the user
3. I've repeated step two with another user and had the same results, and I've seen an ip_abuse entry as well
4. I've sent anonymous requests both from the browser and using the python requests script and the abuse report was not submitted either (there are no activity logs for anonymous requests apparently so I don't have the exact examples)
5. The throttling also applies to user abuse reports

There is something I've noticed though: user_abuse throttles are set for 24 hours while IP throttles seem to last only for ~ 3 hours (I don't have the exact time because I started counting a bit later after the requests were sent and than I was already at a 2.3 hours waiting time).

Hope this helps and if @ioanarusiczki arrives to different results we can compare our findings.

[diox]

Both throttles should last 24 hours. However, something to keep in mind when testing on dev: cache will be reset at each deploy! So depending on what was happening on dev when you were testing that may have had an impact.

[AlexandraMoga]

Both throttles should last 24 hours. However, something to keep in mind when testing on dev: cache will be reset at each deploy! So depending on what was happening on dev when you were testing that may have had an impact.

I was testing on stage

[ioanarusiczki]

@diox @AlexandraMoga

I repeated my testing on AMO stage. I sent using Postman 20 consecutive requests for an addon using a Session id -> after 20 requests with https://addons.allizom.org/api/v5/abuse/report/addon/ I hit the throttle:
"detail": "Request was throttled. Expected available in 83747 seconds."

Then authenticated with the same user I tried to send a user abuse report from FF -> I cannot because I get the "Request was throttled, Expected available in 83657 seconds."
Then I tried to send addon abuse reports. In this case, I can still send them .

I'm wondering if I'm doing something wrong when I set up the abuse reports for -stage env. ? Yet, if I check the admin the addon reports were sent from the browser.

[diox]

Then authenticated with the same user I tried to send a user abuse report from FF -> I cannot because I get the "Request was throttled, Expected available in 83657 seconds."

When sending an abuse report from FF, that shouldn't use the authentication AFAIK. So if you get throttled at this point, this must be because of the IP - meaning everything is working as expected.

Then I tried to send addon abuse reports. In this case, I can still send them .

What do you mean by that exactly ? After waiting ?

[ioanarusiczki]

@diox

What do you mean by that exactly ? After waiting ?

Maybe the gif would help, I'm into the same browser and I hit the 429 with Postman , then I tried from an AMO page.

[diox]

What's the value for extensions.abuseReport.url in your about:config ?

[ioanarusiczki]

@diox https://services.addons.allizom.org/api/v4/abuse/report/addon/
extensions.abuseReport.amoDetailsURL is https://services.addons.allizom.org/api/v4/addons/addon/
extensions.webapi.testing is true (I turned it false by mistake right before making the gif)

I also checked with Browser Toolbox , after I tried again, I see the api responding

[diox]

Ok. Would be interesting to test entirely with postman with services.addons.mozilla.org - trying various things until we can get simpler steps to reproduce that don't involve using the browser.

[ioanarusiczki]

@diox Ok, I'll try with https://services.addons.allizom.org/api/v4/abuse/report/addon/