wpt.fyi status checks are not showing up on WPT repo #1660
Comments
stephenmcgruer
added
bug
|
I would hazard a guess that the error comes from https://github.com/google/go-github/blob/master/github/messages.go#L201 |
|
My suspicion would be that when the app was changed for #1557 , this changed the secret in some way, but I'm not familiar with the flow there so that may be a red herring. |
|
We read the secret from Line 53 in c114bd7
Which reads it from the datastore. |
|
The secret at https://github.com/organizations/web-platform-tests/settings/apps/wpt-fyi and the one stored in the datastore appear to be different. I'm going to try backing up the datastore entry, changing it, and seeing if that helps. |
|
That does not appear to have helped. I am now suspecting that https://github.com/organizations/web-platform-tests/settings/apps/wpt-fyi is not the right secret; so much for the quick fix. |
|
For now, I have restored the previous secret in the datastore. |
|
For the life of me, I cannot figure out who actually sends requests to |
|
Ok, there's a webhook secret stored in https://github.com/organizations/web-platform-tests/settings/apps/wpt-fyi that I missed. Resetting that to the one in the datastore. |
|
Also, I think we've found the cause. Chrome password manager is overriding bits of the settings page, which means that when @Hexcles changed the app name his password manager probably reset the secret too. |
|
From the logs side, things are recovering. Looking for a WPT PR that has the checks showing up. |
|
Thanks @stephenmcgruer for tackling this today! |
|
I believe we're good here, e.g. see web-platform-tests/wpt#20402 Postmortem is well underway too. |
|
This happened again (see web-platform-tests/wpt#20418 (comment)). I checked the app settings page, and for some reason the @Hexcles were you editing the wpt.fyi app again? Also @Hexcles - did our alert trigger? /api/webhook/check was 500-ing like crazy... |
|
No I didn't... That's really weird. I'd really hope there's an auditing
log.
…On Sun., Nov. 24, 2019, 17:31 Stephen McGruer, ***@***.***> wrote:
This happened *again* (see web-platform-tests/wpt#20418 (comment)
<web-platform-tests/wpt#20418 (comment)>).
I checked the app settings page, and for some reason the value of the
webhook secret input was value='value=[secret]'... I've reset it to just
value=[secret] and expect that will fix the 500s.
@Hexcles <https://github.com/Hexcles> were you editing the wpt.fyi app
again?
Also @Hexcles <https://github.com/Hexcles> - did our alert trigger?
/api/webhook/check was 500-ing like crazy...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1660?email_source=notifications&email_token=AAK6ZDGNZULLOZI2BJNDLXDQVL6DBA5CNFSM4JQQFQH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFAWSQA#issuecomment-557934912>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAK6ZDGH3UNVIEIGXF4ZLBDQVL6DBANCNFSM4JQQFQHQ>
.
|
Yes, but apparently Stackdriver failed to send emails to group aliases. Would you mind me adding your email directly, along with mine? |
|
Sgtm. If we are worried about the corp filter blocking it, also happy to add my chromium or even personal email... |
|
I might've got an explanation of what happened: I forgot to save the setting when trying to fix my screw-up last Friday -- remember I told you I accidentally did it again last Friday @stephenmcgruer ? I told you I fixed it, but looking at the open tabs of my workstation, I realized the settings tab was still open so chances are I did not actually click save... Sorry about that. On the bright side, this has been a good test of our alerting setup -- and it didn't work. |
|
There is an audit log for admin activities, but somehow it doesn't include changes to apps... Sigh. https://github.com/organizations/web-platform-tests/settings/audit-log |
|
Oh dear :(. These things happen though, and if I had more control over the app settings I would definitely have included changing that UI as part of the postmortem (adding a confirmation box, for one!). Now that we seem to have alerting working (I got an email saying the outage was over); would we have continued to receive emails if we thought we had fixed it but it was still actually broken? (I.e. are there 'update' emails whilst an outage still continues?) |
|
Yes we should get emails if another outage occurs.
…On Mon., Nov. 25, 2019, 06:23 Stephen McGruer, ***@***.***> wrote:
Oh dear :(. These things happen though, and if I had more control over the
app settings I would definitely have included changing that UI as part of
the postmortem (adding a confirmation box, for one!).
Now that we seem to have alerting working (I got an email saying the
outage was over); would we have continued to receive emails if we thought
we had fixed it but it was still actually broken? (I.e. are there 'update'
emails whilst an outage still continues?)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1660?email_source=notifications&email_token=AAK6ZDEUWQKXT22XVPPTQQ3QVOYTDA5CNFSM4JQQFQH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCB4RQ#issuecomment-558112326>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAK6ZDD2CQPJD227U52WHZ3QVOYTDANCNFSM4JQQFQHQ>
.
|
See web-platform-tests/wpt#20386 (comment)
cc @foolip @Hexcles
Investigating now. I'm seeing 500 errors on
/api/webhook/checkfor prod wpt.fyi, which may be related:2019-11-22 04:42:08.264 PST payload signature check failedThe text was updated successfully, but these errors were encountered: