Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addons FxA webhook endpoint is returning 403s to FxA #9044

Closed
biancadanforth opened this issue Jan 17, 2023 · 21 comments · Fixed by mozilla/addons-server#20210
Closed

Addons FxA webhook endpoint is returning 403s to FxA #9044

biancadanforth opened this issue Jan 17, 2023 · 21 comments · Fixed by mozilla/addons-server#20210
Assignees
Labels
Milestone

Comments

@biancadanforth
Copy link

biancadanforth commented Jan 17, 2023

Describe the problem and steps to reproduce it:

What happened?

Starting November 10th, 2022, the prod Addons endpoint (https://addons.mozilla.org/api/auth/fxa-notification) started throwing 403s for webhook delivery requests from FxA.

What did you expect to happen?

200s from this endpoint.

Anything else we should know?

Per @bbangert:

We make the request in a uniform manner to all RP's.

Since we don't see any other RPs (relying parties) returning the same errors, we are guessing it might be an issue validating the request.

┆Issue is synchronized with this Jira Task

@diox
Copy link
Member

diox commented Jan 18, 2023

Need to trigger password change, primary email change and account deletion from FxA on dev once mozilla/addons-server#20201 is deployed, and look into the logs.

@ioanarusiczki
Copy link

@diox Right now this is open , waiting to be closed.

@diox
Copy link
Member

diox commented Jan 19, 2023

Sorry, still opened because I haven't got a fix yet, just added more logging to see what's going on. You can try triggering events I noted in #9044 and I'll look at the logs.

@ioanarusiczki
Copy link

@diox

From the fxA Profile page (which is Manage Firefox Accounts URL that redirects from AMO's Edit Profile Page to fxA stage)

I changed password and the primary email for rusiczki.ioana+35@gmail.com
Deleted account for rusiczki.ioana+37@gmail. com

@diox
Copy link
Member

diox commented Jan 19, 2023

I don't see anything in the logs, unfortunately this could mean it's not connected correctly for our dev environment... We might need to test on stage (which doesn't have the added logging yet)

@ioanarusiczki
Copy link

@diox Done on -stage , same accounts.

I'll try again -dev with fxA stage, there could be a difference in behaviors but I'm not sure yet.

@diox
Copy link
Member

diox commented Jan 19, 2023

The patch to add necessary logging has hit stage yet (needs to be cherry-picked)

@ioanarusiczki
Copy link

@diox

Right now on -dev when I changed password and left the edit profile page open in a different tab I an not being logged out. Meanwhile on -stage if I refresh the edit profie page -> I am logged out

When I switch the primary email , on dev the old email is still displayed (only after I log out and login again I will see it changed). On stage the email address changes to the new one at a page refresh (no need to logout)

@ioanarusiczki
Copy link

ioanarusiczki commented Jan 19, 2023

@diox

I see the same at deleting the account.

On -dev I am left with a login.
On -stage I'm automatically logged out from AMO.

P.S. these are older -dev issues. Ok, I'll wait for -stage then.

@diox
Copy link
Member

diox commented Jan 19, 2023

The fact that you are automatically logged out/email is automatically updated on stage means that the error isn't reproducible there, so that's interesting...

@ioanarusiczki
Copy link

@diox

I don't see a different behavior on -stage after repeating the testing now.

@diox diox modified the milestones: 2023.01.19, 2023.01.26 Jan 19, 2023
@diox
Copy link
Member

diox commented Jan 19, 2023

I can see some errors on stage & prod now with the added logging: The token is not yet valid (iat). leeway is 0, since this is not re-using the JWT config we have for our developer API keys.

@sentry-io
Copy link

sentry-io bot commented Jan 19, 2023

Sentry issue: ADDONS-SERVER-PROD-B5

@sentry-io
Copy link

sentry-io bot commented Jan 19, 2023

Sentry issue: ADDONS-SERVER-STAGE-35

@diox
Copy link
Member

diox commented Jan 19, 2023

The root of the problem is that FxA is sending iat as floats with millisecond precision. When we upgraded PyJWT, we got this extra check, jpadilla/pyjwt#794, but PyJWT is using the iat provided as is (a float in our case), comparing against the current datetime as an integer. So without a leeway, it fails (almost) everytime.

jpadilla/pyjwt#814 would likely fix this but is not yet deployed. FxA could also send the iat as integer to help, but I think we'll want a short (1 to 5 seconds max - all tracebacks I found didn't need more than 1) leeway just in case.

@diox
Copy link
Member

diox commented Jan 19, 2023

Since QA couldn't find any functional issue, I suspect most of these errors could be caused by notifications of events we don't actually care about... But it'd be good to fix in any case.

@diox
Copy link
Member

diox commented Jan 19, 2023

Fix is on stage.

@ioanarusiczki
Copy link

ioanarusiczki commented Jan 20, 2023

@diox

I made all the necessary operations again. -> it did not change in behavior since yesterday

I've noticed at change password that after I'm logged out of AMO and try to login again, I'm asked to Confirm sign-in on email.

I did not receive the email to confirm the sign-in from the first attempt so I clicked on Resend which works, I'm getting the email. -> this happened 3/3 attempts

not in inbox

@bobsilverberg
Copy link
Contributor

@ioanarusiczki I believe our intention is to push this change (from mozilla/addons-server#20210) to prod today. Has this been verified on stage?

@ioanarusiczki
Copy link

@bobsilverberg The results above are from today's checking.
As I said, I did not see changes compared to yesterday's testing on AMO stage.

@KevinMind
Copy link
Contributor

@KevinMind KevinMind added repository:addons-server Issue relating to addons-server migration:2024 labels May 4, 2024
@KevinMind KevinMind transferred this issue from mozilla/addons-server May 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants