Only run scheduled tests every week

[dstansby]

This reduces the frequency of our scheduled tests from daily to weekly. My motivation here is:

1. We do not respond to test failures on a daily basis (and don't have the developer resources to do so)
2. Running tests every day uses a lot of CI resources. These are £ free (currently), but have an environmental cost in all the energy it takes to run them.
3. I think I would be at least personally be more likely to actually take notice of any failures every week and act on them, compared to every day

[nabobalis]

1. We do not respond to test failures on a daily basis (and don't have the developer resources to do so)

I feel like we do respond, if it isn't the online build.

2. Running tests every day uses a lot of CI resources. These are £ free (currently), but have an
   environmental cost in all the energy it takes to run them.

I think we should only worry about it, if/when we start paying for CI resources.

[ayshih]

As a counterpoint, running scheduled tests every day does make it quicker to isolate when a new release in our web of dependencies is causing a problem. Speed can be particularly important to be usefully responsive to a user that suddenly has a problem with a fresh install.

Is it possible to set up some magic with the scheduled tests so that they are cancelled if there are no new commits in the repo and no change at all to the build environment (i.e., all dependency versions are the same as the last time)?

[nabobalis]

Is it possible to set up some magic with the scheduled tests so that they are cancelled if there are no new commits in the repo and no change at all to the build environment (i.e., all dependency versions are the same as the last time)?

It used to be simple to enable if no new commits on Azure, I am unsure about Actions.

The build environment changing I imagine would need something custom.

[ayshih]

Actually, thinking about it more, you'd want to run scheduled tests even with no new commits. They're really for testing for a change in dependencies or a change in server response. For that latter possibility, there's no magic that we can write for that, so I think that the online test should be run daily regardless.

[nabobalis]

We could reduce the number of jobs we run daily to be only:

1. Online (as it runs all offline tests too)
2. devdeps
3. doc + doc gallery

That should cover everything?

[ayshih]

Sure, that'd work

[ConorMacBride]

Maybe oldestdeps also?

[nabobalis]

Ah yes, I forgot about that

[dstansby]

As a counterpoint, running scheduled tests every day does make it quicker to isolate when a new release in our web of dependencies is causing a problem. Speed can be particularly important to be usefully responsive to a user that suddenly has a problem with a fresh install.

Is this a regular issue? I can't remember it happening recently, and when it does happen I think package versions don't change that often that the difference in installed dependency versions on a weekly timescale would be too large to track an issue down to a given package (I'd say this is getting into tricky territory on monthly timescales).

I'd also add that given we only release bugfixes once per month, having a ~week latency on spotting bugs compared to a ~day latency isn't a major issue.

[nabobalis]

Is this a regular issue? I can't remember it happening recently, and when it does happen I think package versions don't change that often that the difference in installed dependency versions on a weekly timescale would be too large to track an issue down to a given package (I'd say this is getting into tricky territory on monthly timescales).

We had this with numpy last week.

I'd also add that given we only release bugfixes once per month, having a ~week latency on spotting bugs compared to a ~day latency isn't a major issue.

We should be doing bug fixes sooner if a big enough problem has cropped up.

[dstansby]

We had this with numpy last week.

I forgot that... but I think it illustrates my point - given it took 8 days to turnaround a bugfix, and that's still not in a release, I'd say we don't have the development capacity or need to act on a daily timescale to issues.

We should be doing bug fixes sooner if a big enough problem has cropped up.

Why? Is there demand from our users? Do we even have the developer capacity to do this?

[nabobalis]

We had this with numpy last week.

I forgot that... but I think it illustrates my point - given it took 8 days to turnaround a bugfix, and that's still not in a release, I'd say we don't have the development capacity or need to act on a daily timescale to issues.

It could be in a release, it really should be out sooner rather than later.
We already had a user hit this problem and report it on discourse.

Why? Is there demand from our users? Do we even have the developer capacity to do this?

We are trying to have a functional project and ensuring that our releases aren't broken kind of feels important to me?

The demand is from me and yes we have that capacity.

[dstansby]

How about we meet halfway, and run the CRON jobs 3 or 4 days a week? Still gives us a reasonable turnaround on finding bugs, but halves the amount of CI time spent on CRON jobs.

[nabobalis]

We could cut down number of cronjobs but run them daily?

Online (as it runs all offline tests too)
devdeps
doc + doc gallery
oldest deps

That cuts it down from the 15 jobs (counting the badges on the element) to 5?

We could run the full set once a week (for releases and extra items).

[dstansby]

I still reckon running every other day (Monday/Wednesday/Friday) would be fine.

I'm 👍 to reducing the number of runs in the regular runs. We should probalby work out what the purpose of the regular run is - if it's just to catch bugs caused by dependencies in a timeley manner, then we could probably just do:

• Online
• devdeps
• oldestdeps

[Cadair]

There are multiple things that the scheduled jobs provide:

• Regular runs on the main branch, with notifications, so that we see if something has broken (this applies to all jobs we run on PRs or not)
• Testing some more esoteric configurations we support but do not need to burn CPU time running for every PR or merge.
• Ensuring regular testing of upstream changes in our dependencies for new releases and dev changes.
• Testing that the VSO and JSOC are still online 😜

I am more in favour of keeping all the jobs we currently run and reducing the frequency to Mon/Wed/Fri than cutting the builds, as I think there is value in running everything on a regular schedule.

[nabobalis]

I am more in favour of keeping all the jobs we currently run and reducing the frequency to Mon/Wed/Fri than cutting the builds, as I think there is value in running everything on a regular schedule.

Are you in favour of cutting them down at all?

[ayshih]

My feeling is that:

• Ensuring regular testing of upstream changes in our dependencies for new releases and dev changes.
• Testing that the VSO and JSOC are still online 😜

are tests we get tangible benefit from running daily, even with only a limited set of builds, while:

• Testing some more esoteric configurations we support but do not need to burn CPU time running for every PR or merge.

is something we can do less frequently, every couple of days, if not even less frequent. In other words, let's not throw away any of the currently scheduled builds, but I want a subset to continue to be run daily.

[nabobalis]

is something we can do less frequently, every couple of days, if not even less frequent. In other words, let's not throw away any of the currently scheduled builds, but I want a subset to continue to be run daily.

These builds tend to be quite short, so running them everyday isn't really a problem.

[nabobalis]

My concern is making the CI configuration more complex for no tangible benefits.

I personally think we should not change anything overall.
I would like to make the message in matrix only tag/label the failed test runs instead of all.

[dstansby]

I'm going to close this since there's no easy consensus here.