Split out CLI version tests to separate workflow

[charisk]

We currently run all CLI integration tests for every supported version of the CodeQL CLI, for windows and linux and for every commit of every PR. This is wasteful and it also adds minutes and chance of hitting flakiness.

We've also not generally seen tests succeed on one version and fail on another (outside of flakes), so pragmatically it doesn't make a huge amount of sense to run these tests so often. There aren't a lot of code changes that are related to different CLI versions.

This PR changes the workflows so that instead of running the whole matrix every time, we:

• Run only against the latest released CLI version for PR workflows
• We have a daily run of the CLI version tests

If we're finding that we're missing legit failures on the new workflow, we can add a process to notify the team when a failure happens. I'm thinking that at the moment it's worth doing this as it may be too noisy (due to flakiness).

You can see the new workflow here: https://github.com/github/vscode-codeql/actions/runs/5923448364

Checklist

N/A:

• CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
• Issues have been created for any UI or other user-facing changes made by this pull request.
• [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

[robertbrignull]

We'll need to remove all the branch protection rules before we can merge this. Which is also a step that'll need to be removed from the CodeQL CLI release process docs.

But fun fact, you can see from "13 successful and 13 expected checks" that this PR reduces the number of CI workflow runs by 50% 😌

[robertbrignull]

LGTM

Do we want/need more review on this from the CodeQL team, or are you happy to just go ahead?

[robertbrignull]

A thought, that I didn't think of before approving: don't include this in the matrix here. This will make the name of the CI job just CLI Test (ubuntu-latest) instead of CLI Test (ubuntu-latest, v2.14.2) and it will mean the job name won't keep changing over time as the CLI gets updated. This means we won't have to update the branch protection rules every time the CLI version changes.

[charisk]

Good shout!

[charisk]

Apologies I should have mentioned the plan in the PR description (because even I forgot about it for a minute 🤦 !). I was thinking of the following:

• Get PR reviewed from our team (✅) to make sure we're on the same page
• Check CodeQL experiences folks are happy with this change and create PRs to update the various docs on this
• Remove protection rules
• Merge PR

[aeisenberg]

It's probably fine as it is, but I just want to be sure. Feel free to merge if the answers to my questions are both no.

[charisk]

@robertbrignull let me know if you're still happy with this PR to go ahead

[robertbrignull]

I'm still happy with this 👍🏼