Check seems to be stalling lately (GitHub API checks.create slowness) #25
Comments
|
Hi @tir38 I've been noticing that today again as well. |
|
Update: Looks like the Github issue has resolved now - https://www.githubstatus.com/incidents/fpk08rxnqjz2 |
|
The issue is still happening for me – I've tried checking/unchecking and re-added the status check to PR's. |
|
Hi @jlai403 I've checked the server the check runs on and it's running fine, no errors and I can see requests hitting it (though I purposefully don't log any details about the requests for security). Interestingly, for me at least, in github it's showing for one of my PR's that the check has completed but that github is for some reason still awaiting checks? If it doesn't resolve in an hour I'll report it back to Github :) |
|
@jlai403 is it running faster for you now? My checks seem to be running faster, though I'm still seeing the "Required statuses must pass before merging" issue. If your issue was more of the stalling or the checks taking forever, could you try again now as github appear to be working on this again and it's running the checks themselves faster. |
|
@stilliard – appreciate the quick response. I've updated our Branch Protection rules and re-added the application – seemed to resolve the issue for now. Thanks! |
|
Awesome, glad it's working well again for you. I'll keep this ticket open and will report back when Github get back to me too. |
|
Hi! It's getting stuck quite often for us and GitHub doesn't seem to be having problems (at least not right now). Do you think re-adding the app can help? Edit: not entirely sure if this happens every time, but I noticed it's stalling after a force push to a branch when rebasing it. |
|
Hi @vamcs Might be worth trying though normally editing the name or body of the PR will help send it over to us again. From my debugging, normally github send it to us quickly, but then we make several API requests back to github to check the comments, review comments & diff comments of the review, lately I've noticed these can at random times take a while for github to respond on. My checks seem to be running ok for me atm but could be an intermittent issue. |
|
Just as an update for the original issue, github replied last week about the ticket I'd submitted and confirmed the api issues they had at the time were likely the cause.
|
|
We have also been experiencing this issue. In the case of the screenshot below, this PR is approved and all tasks are checked. Removing a task or re-checking it does not seem to "kick" it. Is there a workaround in place? Perhaps merging the main brach back into the PR will re-check the task status?
|
|
Hi @prtcl I've not had this since the last issues a couple weeks back, looks like github's not reporting any issues.
Here's some output from the log showing a couple PR's coming in and completing the checks in a couple seconds, watching them atm they seem to be all running ok. Can you tell me the pull request ID/number for the one that's stuck atm please? I'm now logging the ID as that gives away nothing about the repo etc. as it's an incremental ID for each repo. Thanks. |
|
@stilliard It's happening to me as well now. Do you want me to open a new issue? Thanks! |
|
Hi @adar-h-healthy |
|
I get this a lot too. Is there any way to force the check to re-evaluate and/or see the status of the api calls? I sit there unchecky checky and I don't know if it's having any affect :D |
|
Hi @dave-yotta |
|
I was waiting on #393 and #1833 but merged in the end without them. Didn't realise this was an external service, makes sense it would get slow if you're trying to scale with the whole of github. |
|
Thanks for sending those over. Here's the logs for those ID's from the last 3 hours: (UK timezone) Looks like both were sent to us, we checked the PR and it's comments (using the 3 api endpoints for comments) and then the done call is when we sent back to github to say if it's all ok or not. No warnings or errors in the logs around these times either. I think I need to raise another ticket with Github about this as for I've seen this a number of times now where the checks are hitting the server fine, checks running quick and we respond back to say it's done fast too but then no update on the Github UI itself. |
|
Ok yeah - I gave it probably 30-60 seconds and some hard refreshes etc, but didn't see an update. the "waiting for checks" time is usually very long like: (since last click on a checkbox?) but I guess that's just how it's set up, instead of just running for a couple mins each time a checkbox is ticked. |
|
The waiting for checks time is a funny one, essentially when you submit a PR or comment on one, it sends us a check. What's odd is it seems like github's either not getting these back all the time, though from our side they appear to be sending to them fine. Or github are queuing up these responses and not getting to ours quickly to update the UI. |
|
Hmm, comparing ours to the WIP check which works similarly. They have a line about setting the head_branch to blank (https://github.com/wip/app/blob/152a3d261eacfe487db7661b390b1bb718e072a0/lib/free/set-status.js#L11) I'll investigate these and see if maybe we need to modify our request back to help fix these issues. |
|
I've made the changes, still seems to be a little slow but my test request completed on the server in under 1second but github took over 2 and a half minutes to update the UI. They have now however put a notice on their status page about issues with PR's so it's possible the delays we've seen again today are to do with this: |
|
Hey, To help I've made a couple tweaks to retry requests based on how the WIP check works. Hopefully the combination works well, testing on a couple repo's the updates run on the server side in less than a second still but on the github UI they are updating in a few seconds too. If there's any further issues with checks not completing then please check to see if github is having issues over on https://www.githubstatus.com/ and if nothing is reported please either comment here or start a new issue if needed. |
|
Thanks for your work, @stilliard. |
|
No worries, glad it seems resolved now :) |
|
we at osu-wiki still run into this every few weeks or so, and the check takes ~15-20m to complete, or even more: |
|
Hi @TicClick |
|
Here's a full log from our app for that first PR 7379 for example: In the logs output you'll see each check start with "Request received", then 3 api lookups for comments on the PR, and finally "Complete and sending back to github" when we send back to Github to say it's finished:
Thanks for providing those PR links/IDs, that's at least helped us double check the app is running the checks themselves fast, just something delaying github either receiving our check response, or them queuing it up and taking a while to update the UI? |
|
hi, it happens irregularly (didn't occur today as far as I remember). wouldn't be surprised if it's github's events being ostensibly -- and silently, without the status page update -- slow. thanks for the investigation. |
|
No problem, if it happens again let me know with a PR link again & I'll try to report it quickly to them, maybe GitHub support can share some light on what's happening. |
|
I've pushed a new update which includes better logging for Github to help debug this issue (as well as a couple extra features for optional and skippable tasks, see readme). Please let me know if you see any further issues and what your repo & pr ID is, this should hopefully help us track down the cause. |
|
Will keep an eye out, thanks. |
|
Since adding this I've seen a few requests running slow today so opened a new ticket with github and reported over to them with the full logs. |
|
GitHub support replied to say they haven't had chance to look into it yet but will hopefully follow up with it next week. Also I've changed the timeouts & retries to hopefully help resolve this as well: |
stilliard
changed the title
|
another stalled pr ppy/osu-wiki#10269 (took 14 minutes) |
|
Hi @Walavouchey No reply back from Github yet sorry. Here's the logs for your check:
all logs show it was good and we sent to github, but no reply from them yet. |
|
Until I hit send on that comment aha, good timing from github.
|
|
ppy/osu-wiki#10268 stuck for 40 minutes atm |
|
No reply from github yet, will update when I hear back from them. One thing to try, make a quick change to the pr title or body, or untick then re-tick. |
|
yup, we do that often hoping things will improve, but it's still a game of
chance — more often than not it still takes dozens of minutes for edits to
do something, if anything at all. recent example:
ppy/osu-wiki#10194
…On Wed, 25 Oct 2023 at 18:31, Andrew Stilliard ***@***.***> wrote:
No reply from github yet, will update when I hear back from them.
One thing to try, make a quick change to the pr title or body, or untick
then re-tick.
In cases where it's been stuck a long time this can mean the following
request completes before the original.
Hope this helps
—
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABDYCFMXC4RMLW3FAJ6RL6TYBE5G7AVCNFSM5Q4OQKEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXHE3DIOJQGUZQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Yeah it can be frustrating I understand, unfortunately without a reply from GitHub on it there's not much we can do as it appears to stall their side receiving in the check create call. |
|
Another option btw is you could try self hosting it. Or if you want, there's a github action based flow here: https://github.com/Shopify/task-list-checker/ Will report back when I hear anything from github still though. If anyone has a contact at GitHub and wants to talk to them about this, our ticket is #2383169. |
|
I have a plan, off the idea about it possibly being traffic from our ip to github backing up. |
|
Had some time tonight to play around with the new idea. I've set up a new load balancer and a couple nodes for now (may add more as needed) dedicated to this app. These are now in San Francisco, which may perform better than the uk location previously on DO. Monitoring it atm and it seems to be running fine however this time of day is often has less traffic for the service anyway, the real test will be tomorrow onwards. Let me know if you notice any issues, also worst case I can switch it back to the old setup with a single url setting change in github just in case as well. |
|
Update since moving the servers: |
|
Anecdotally, they seem to be working much better now. I had a moment of surprise when a PR went green right after clicking a checkbox which never happened before. |
|
@dave-yotta |
|
Been a couple weeks and not had any further issues. |
|
I just noticed this one today, could you double-check? ppy/osu-queue-score-statistics#164 |
|
@peppy thanks for reporting. This one is interesting, I don't see it at all in the log atm. I wonder if Githubs not sent it yet for some reason? |
|
Nothing reported on the github status atm. |
|
Editing did help in this case. |
|
Cool, checked the logs and I can see 2 updates just came in for your repo. |
I've noticed this a few times. Checking a box triggers a check and I end up in this limbo state where the check never completes
task-list-completed Expected — Waiting for status to be reportedChecking/unchecking/adding comments doesn't seem to retrigger anything. I can share a repo if that would help.
The text was updated successfully, but these errors were encountered: