Daily builds seem to have stopped

[kcarlson]

What version of gRPC and what language are you using?

3.14

What operating system (Linux, Windows,...) and version?

Linux

What runtime / compiler are you using (e.g. python version or version of gcc)

n/a

What did you do?

Navigated to https://packages.grpc.io/

What did you expect to see?

Todays daily build

What did you see instead?

Last daily build was on 2020-12-02

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

[drfloob]

Thank you for pointing this out. I can confirm our nightly builds have been failing since 12/2. I'll look into the fix, and better monitoring for this problem.

[drfloob]

The nightly build is fixed, and I've improved our internal monitoring for this build job. The script did not backfill nightly builds between 12/2 and 1/20, and I noticed the build has been pretty spotty in the past as well (there are only entries for half of Nov 2020). I think we'll live with the missing artifacts for now, and keep a better eye on these build failures going forward. Thanks again.

[Falco20019]

@drfloob I wanted to have a nightly build for #25654 and saw that it's still pretty spotty. Only 2 out of 10 days in March generated builds, but there were also new commits every day (besides March 7th). Even on those days, I would have expected to be a build for the same commit, like to be seen with 2021-02-28 and 2021-03-01. So you might want to reopen this and have another look. I can also create a new issue if you would prefer.

[drfloob]

Thanks @Falco20019, it is a known problem. Our nightly artifact builds have been timing out due to gRPC's growth in the past few months. I'll see about increasing the build timeouts again as a stopgap measure, but we'll need to find a more sustainable solution. Apologies for the inconvenience.

[nahk-ivanov]

Seems like builds stopped again, the last build I see there now is from 2022-09-27. Bump the timeout yet again?