Speed up, possibly pare down tests

[rfay]

Is your feature request related to a problem? Please describe.

Tests are taking a really long time to run. Some of the test runs on Windows take more than 2 hours. However, the new arrangement of travis-ci tests, with some parallelization, finishes faster than anything else at this point.

Describe the solution you'd like

• Run some of the testpkg and testcmd CircleCI tests in parallel, as is done in the travisci.
• Run the container builds and tests in parallel, instead of running them all sequentially
• Figure out how to run more relevant tests for certain PRs
• Consider a way to rerun only tests in the area of a failure; often failures are due to external dependencies, and it would be nice to just run those again.

Anyway, we spend a lot of time and effort and even money on the tests, and it would be nice for them to complete a little sooner and use less resources.

[rfay]

To reduce usage of our buildkite test runners, we can

• Create a dynamic pipeline which only actually executes if golang code has changed
• Stop using so many buildkite permutations. Just do mac NFS and Windows NFS (and later we'll have WSL2)

The feature request to buildkite for a path filter is buildkite/feedback#243

[rfay]

I turned off all the macOS and Windows tests except one each for the full tests under NFS. We expect people to be using NFS anyway, and so we might as well focus our tests there.

[AronNovak]

@rfay Is there anything we can do in order to have green status from the CI overall? Otherwise sane PRs are red due to BuildKite.
Should we try the dynamic pipeline route that you suggested above?

[rfay]

Mostly, the problem is not buildkite, but Docker Desktop, which is flaky in Windows and especially on macOS arm64. It's also where people do most of their work. If you have a specific problem or worry I can help with it. I look at every test situation and retest if I don't know the flaky failure intimately. On important PRs I normally get everything green before pulling. I can easily grant you retest privs on buildkite if you have a specific worry.

[rfay]

I happened to examine a Windows NFS buildkite build and there were some amazing offenders.

TestDdevXdebugEnabled takes more than 11 minutes, TestMutagenSimple takes 17m.

TestComposerVersion 103.60s
TestCmdSnapshotRestore 104.56s
TestHttpsRedirection 109.17s
TestCmdPauseContainers 115.75s
TestPoweroffOnNewVersion 116.53s
TestPortOverride 121.46s
TestGetFreePort 126.91s
TestWebserverType 134.79s
TestDdevFullSiteSetup 137.89s
TestExtraPackages 150.52s
TestDdevImportDB 159.11s
TestCmdMutagen 164.03s
TestDdevRestoreSnapshot 192.21s
TestDdevXhprofEnabled 248.60s
TestDdevAllDatabases 267.51s
TestConfigMariaDBVersion 414.67s
TestDdevStart 454.97s
TestDdevStartUnmanagedSettings 471.01s
TestComposer 560.36s
TestDdevXdebugEnabled 683.87s
TestMutagenSimple 1056.15s

Notes on ways to process:

awk '{gsub(/[\(\)]/, ""); print $3, $4}' <list.txt
sort +1.0 -n

[rfay]

I'm going to close this for now, as a number of things have been addressed and there's always more room for work on this. It's an ongoing priority, not an issue. But the recent improvements are good.

• Buildkite and circleci don't waste time on docs now
• The github actions tests are more focused