404 when clicking on "pending completion"

[jrfnl]

The "pending completion" link goes to https://docs.coveralls.io/parallel-build-webhook which shows a 404.
The mouse-over for the question mark is also not working.
And I get the impression that clicking the "Set Done" button has no effect whatsoever.

[jrfnl]

And I get the impression that clicking the "Set Done" button has no effect whatsoever.

I wonder if this part of the issue may be due to some of these builds having run into the outage of this weekend.
Previously, I'd already "Re-run failed jobs only" for those builds, but that didn't help.
I'll be restarting those builds from scratch now to see if that makes a difference.

[afinetooth]

Hi @jrfnl. Thanks for the heads up. Yeah, there shouldn't be a link under "pending completion"---or else it should hit the same URL as SET DONE, which hits a utility endpoint called /finalize that tries to finish your build under the presumption it was stuck.

I'll submit a ticket to fix the "pending completion" link.

In the meantime, let me get a closer look at why you're seeing pending builds in PHP_CodeSniffer. I'll push them through if I can. Will update you here as soon as I'm able.

[afinetooth]

Hi @jrfnl. Quick update on those builds:

• All completed, as I'm sure you're aware; I understand you re-ran those, but based on the original creation timestamps I see reasons for them being long-running, and the pending states, and can say that, under those circumstances, the pending states would have been temporary, regardless of how long-running the builds ended up being:
  • From the timestamp on your example build (my view below is showing PDT times) that build came through right at the beginning of our (long) weekend maintenance window;
  • Then, if any came through in the next 24 hrs, they would have been similarly blocked/delayed. No jobs were processed during that maintenance window.
  • And then, in terms of those eventually being processed, they should have had time to process in the ~7 hrs between 4:00pm PDT, Sun, Apr 6 and 10:45pm PDT, Sun, Apr 6, when we had this incident, which could have delayed any unfinished builds at that point for an additional 3 hrs.
  • From that point, if any experienced further delays, it would have been from competing traffic as backed up queues were cleared.

The point being, bad timing on those builds since they conflicted with our maintenance window, and the incident that occurred Sun night after that window, which was caused by an oversight during a cutover procedure during that maintenance window.

The good news is that no similar issues should occur going forward. We are now just performing some database tuning (rebuilding indexes) that should speed up slower builds over the next few days.

Thanks as always for the report and your patience as we complete our infrastructure upgrade here.

[jrfnl]

Thanks for the update @afinetooth ! I became aware of the maintenance quite quickly when it started, the status page told me what was going on, so I wasn't concerned about it. Those things sometimes just need to happen.

My concern was more related to what I believe you refer to as the second incident - where after the maintenance was finished, builds were backing up and not finishing, even after that second incident had been resolved.

Happy to hear that the prognoses for the future is positive though. Well done you all on those infrastructure upgrades 🙌🏻

So what remains is really the original issue I reported above - the 404 on "pending completion" (though without any pending builds for my projects at this time, I cannot reproduce this nor verify whether it has been fixed 😄 ).