Two crates stuck building for multiple days?

[acotis]

Hi, I don't know for sure that this is a problem, but two crates have been in the "currently being built" queue for multiple days (at least since two days ago, I think) and it's causing other queued items to build up (currently over 1000 of them):

I noticed this because my own crate has been stuck in the queue for many hours, and I thought it might be abnormal because the other day when I published an earlier version of this crate, it got through in well under an hour and there were far fewer items in the queue, and because when I click through to either of the two stuck builds, they both say they're "in the build queue" and that their computation time should be limited to 15 minutes (which is clearly not the case).

[syphar]

Yeah, there is a continious peak in releases, especially some workspaces with multiple crates that lead to this full queue.

The two build servers are working normal speed (~25 builds per hour per worker).

I'll check if there is something wrong with these two in-progress crates, the marker is sometimes not 100% correct.

[syphar]

The two helios- crates in progress were a remnant of the server reboot when docs.rs was down,

the build server was still reportitly slowly working on the queue, while new releases were queued continously.

Typically the queue cleans up after a couple of days, we are watching it.

Sadly with the old infrastructure a new build-server is a bit of work, involving another docs.rs downtime, but we're already thinking about it

[syphar]

looking at the metrics, since the downtime on the weekend, only one builder was correctly working.

[syphar]

So that was the reason,

I'll close this issue, thanks for the ping!