GH Actions: run the tests twice - once with cache, once without

Adjust the GH actions test workflows to run the complete test suite once with the cache turned on and once with the cache turned off. The test run with caching turned on will run the tests twice and only run the tests which don't need isolation. This should safeguard that the cache does not negatively influence sniff results. The code coverage job will (should) run with caching turned _off_ to prevent tests not reaching all code paths due to cached results from previous tests short-circuiting things. Note: the effect of caching on the test suite will be minimal as most tests will only execute a function once for a particular input/code snippet. Still good to safeguard/double-check though.
PHPCSStandards · Jun 30, 2022 · ecfef72 · ecfef72
1 parent 7f6443b
commit ecfef72
Show file tree

Hide file tree

Showing 2 changed files with 21 additions and 4 deletions.
diff --git a/.github/workflows/quicktest.yml b/.github/workflows/quicktest.yml
@@ -74,7 +74,14 @@ jobs:
         if: matrix.phpcs_version == 'dev-master'
         run: composer lint
 
-      - name: Run the unit tests
+      - name: Run the unit tests without caching
         run: vendor/bin/phpunit --no-coverage
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: false
+
+      - name: Run the unit tests with caching
+        run: vendor/bin/phpunit --testsuite PHPCSUtils --no-coverage --repeat 2
+        env:
+          PHPCS_VERSION: ${{ matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: true
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -189,11 +189,19 @@ jobs:
         with:
           composer-options: --ignore-platform-reqs
 
-      - name: Run the unit tests (non-risky)
+      - name: Run the unit tests without caching (non-risky)
         if: matrix.risky == false
         run: vendor/bin/phpunit --no-coverage
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: false
+
+      - name: Run the unit tests with caching (non-risky)
+        if: matrix.risky == false
+        run: vendor/bin/phpunit --testsuite PHPCSUtils --no-coverage --repeat 2
+        env:
+          PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: true
 
       - name: Run the unit tests (risky)
         if: ${{ matrix.risky }}
@@ -281,17 +289,19 @@ jobs:
         if: ${{ steps.phpunit_version.outputs.VERSION >= '9.3' }}
         run: vendor/bin/phpunit --coverage-cache ./build/phpunit-cache --warm-coverage-cache
 
-      - name: "Run the unit tests with code coverage (PHPUnit < 9.3)"
+      - name: "Run the unit tests without caching with code coverage (PHPUnit < 9.3)"
         if: ${{ steps.phpunit_version.outputs.VERSION < '9.3' }}
         run: vendor/bin/phpunit
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: false
 
-      - name: "Run the unit tests with code coverage (PHPUnit 9.3+)"
+      - name: "Run the unit tests without caching with code coverage (PHPUnit 9.3+)"
         if: ${{ steps.phpunit_version.outputs.VERSION >= '9.3' }}
         run: vendor/bin/phpunit --coverage-cache ./build/phpunit-cache
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version }}
+          PHPCSUTILS_USE_CACHE: false
 
       # Uploading the results with PHP Coveralls v1 won't work from GH Actions, so switch the PHP version.
       - name: Switch to PHP 7.4