Severe delays in reporting back and status page looks to be stalled

[jrfnl]

Looks like something broken again, as builds don't show up on the overview page and when going directly to the build page (via the link in the GH Actions workflow), the build shows as "pending".

Build #15582818462 is missing:

Direct link from GH Actions workflow: https://coveralls.io/builds/74095541:

[jrfnl]

Hmm... looks like some action was taken, but the issue appears to have come back... (and the backlog is still not fully cleared - above build is now showing in the overview page, but is still at "pending completion"):

[afinetooth]

Hi @jrfnl. thanks for reporting.

I can't get this build to complete yet, but I'm not sure why:

None of the usual suspects.

Will update you.

[jrfnl]

@afinetooth Thanks for responding! We can forget that build - it has been superseded in the mean time - the current build for the open PR is: https://coveralls.io/builds/74109589 (which again, doesn't show up on the overview page and shows "pending completion").

[afinetooth]

@jrfnl OK, thanks. That build has completed.

However, we're struggling again with latency—our servers that run those dequeue samples went offline for the second time this week. We're investigating.

[jrfnl]

@afinetooth Thanks for the update. Hope you all will find a more permanent fix for the servers going down soon, the past few weeks must have been pretty stressful for you all...

[jrfnl]

Closing as the incident looks to have been fully resolved by now.

[jrfnl]

Reopening as the issue appears to be back /cc @afinetooth

[afinetooth]

@jrfnl thanks. We're tracking that.

FYI, that CALCULATION JOB DEQUEUE graph samples from only one (1) of five (5) servers that process our background jobs. Unfortunately, it's a server our devs tend to use to run overnight tasks—like the one that killed it at 3:19am PDT.

So the "good news" is that, in this case, the gap only represented 20% capacity.

But the bad news is, we can't know how much capacity a gap represents, and even a 20% outage clearly put significant upward pressure on latencies as you can see from the graph data when it restored.

I've asked if we can improve the sampling, but unfortunately it will have to wait until we resolve our general latency issues.

I'll update you here, when that's been resolved.

[jrfnl]

Thanks for the update @afinetooth !

Appears the repo I was working in was part of the 20% (Coveralls wasn't reporting back on PRs, which is why I went to the status page).

I've asked if we can improve the sampling, but unfortunately it will have to wait until we resolve our general latency issues.

Sounds like the correct order of priorities ;-)

[afinetooth]

@jrfnl Ok. We're about to deploy a fix.

I will check on your recent builds right after that and reprocess any that seem hung.

[jrfnl]

@afinetooth I'll keep my fingers crossed this is the last hurdle for you all and you'll have a few months without panic ahead of you ;-)

I don't think I have any stalled builds anymore at the moment.

[afinetooth]

@jrfnl Thank you.

I don't think I have any stalled builds anymore at the moment.

Good to know, thanks for the update.