Move visual studio tests from per-release to per-PR

[senhuang42]

Tests for visual studio 2008, 2010, 2012, and 2013 on appveyor were running only on the release branch. Github Actions only supports Visual Studio 2015 and beyond, so we have to preserve these tests.

This moves over the tests to run on branches against dev, and removes 2008 and 2010 tests, since those would exceed the previously discussed 10-year time window for compiler support.

Note that these Visual Studio build tests seem to complain more than the Github Actions ones, picking up on an unused local function issue, since Visual Studio doesn't have an equivalent of __attribute__(unused).

This should get merged after #2846

[Cyan4973]

Let's see after #2846 .
We don't want to inflate test duration on appveyor too much.
We may have to settle on a "good enough" middle ground to avoid making this task an annoying CI bottleneck.

[senhuang42]

I've reduced the length of the tests, and also removed the VS 2013 test, so now we only have a VS 2012 test (having both seems redundant). So now, we test VS 2012, 2015, and 2019 - which seems like good enough coverage.

Looking at appveyor.yml, I realize that the github actions tests don't perform some of these basic fuzz tests. In a follow-up, I'll add that for the VS 2015, 2019 tests as well.

[Cyan4973]

It seems I don't see any Appveyor test running ?

[senhuang42]

It seems I don't see any Appveyor test running ?

Not sure what happened there, but I re-force pushed and it seems to have run and passed this time.

[senhuang42]

Though, I will say that the Release builds take 15-20mins each to build, which is about 10x as long as the GH Actions release builds take. But there's no way to migrate these off of Appveyor, since GH Actions doesn't have MSVC 2012. The options appear to be 1) eat this 40m appveyor cost or 2) Drop the 2012 release builds entirely as not much seems to still support MSVC 2012. (and moving these 2012 release builds to per-release brings us back to the original issue, so it's a non-option).

[Cyan4973]

Indeed, Appveyor tests are now at 38mn (!),
making them by far the slowest element of CI.

In details, I notice that debug visual tests cost only ~1mn each, which is fine,
while release tests are 15-20mn each,
suggesting a difference in scope.

Maybe we should reduce the scope of release tests, or even consider removing them entirely.
I presume debug tests already cover build issues,
which means release tests are mostly present for runtime issues.
Do we need those ?

[senhuang42]

Maybe we should reduce the scope of release tests, or even consider removing them entirely. I presume debug tests already cover build issues, which means release tests are mostly present for runtime issues. Do we need those ?

It's probably good to still have a few runtime tests for windows. It will be better to remove these from Appveyor, and add a new long GH actions test that runs the useful ones (some of these tests look not very useful).

[Cyan4973]

With latest decrease to test durations, with Release runtime tests in the ~2mn range, I believe this PR looks in good enough shape.