Run system tests on rennovate PR's proposing major version updates

[grayside]

Issue #1759 and the cluster addressed by PR #1680 were the result of merging a dependency update carrying a major version bump, which proceeded to break the system tests on run/logging-manual.

Would it be possible to run system tests on major version updates? This is creating escalated P1 bug work instead of P2 internal cleanup work.

[ace-n]

N.B: renovate PRs can typically have higher latency than human-written ones (in part because they're rarely, if ever, urgent) - so the attendant slowdown due to system tests is probably more palatable even if we run system tests on all renovate PRs.

[fhinkel]

Yes, we should run all tests on presubmit, just need to enablet those...
Anybody have spare cycles to add those test configs?

[ace-n]

To be clear - @fhinkel you're saying we should run system tests on all PRs, or only all renovate ones?

(IIRC we may have disabled "system tests on all PRs" in the past because they were too slow. My argument is that that's less important for renovate PRs than human-generated ones.)

[grayside]

Python is taking the approach of running Cloud Run system tests on every PR that touches it. Cloud Run build & deploy is generally 1-2 minutes overhead.

[fhinkel]

I'm in favor of running system tests on all PRs. But we have severa failing P1 system tests, so I can't turn them on on PRs. cc @JustinBeckwith

[fhinkel]

(IIRC we may have disabled "system tests on all PRs" in the past because they were too slow.

I don't remember running systems tests on PRs since moving to Kokoro.

[JustinBeckwith]

With nodejs client libraries we run system tests on presubmit. If it's reasonable to do (I understand it isn't always), that would be my preference. If we have failing system tests, shouldn't those be getting picked up by build-cop-bot? I'm far more concerned that we have failing tests.

[grayside]

Do we have a decision on how to move forward here?

[fhinkel]

Oh, I see we don't have open issues about system tests, now would be a good time to turn them on for PRs. Let me work on that!

[fhinkel]

For run, we always run the system-tests, whether pre-submit or nightly release, see

nodejs-docs-samples/.kokoro/build-with-run.sh

Line 95 in 752c8eb

npm run --if-present system-test

The difference between release and presubmit are the appengine with deployment tests. We decided to not turn them on for presubmit as they take time.

For all other projects with should run system-tests on CI. But I suspect a lot of them are broken by now/never worked in the first place.

[fhinkel]

Fixed by #2118