Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/QA: start recording code coverage #338

Merged
merged 1 commit into from
May 1, 2023

Conversation

jrfnl
Copy link
Contributor

@jrfnl jrfnl commented May 1, 2023

Summary

This PR can be summarized in the following changelog entry:

  • Enable test code coverage monitoring.

Relevant technical choices:

This commit makes the necessary changes to start recording and uploading code coverage data to Coveralls.

Includes:

  • Updating the GH Actions test worflow to run the high/low PHP version builds with code coverage and upload the results.
  • Updating the PHPUnit config to show an inline code coverage summary and provide the logging needed for Coveralls.
  • Ignoring the potentially generated code coverage files in the build/* directory to prevent it being committed.
  • Adding a script to the composer.json file to run code coverage.
  • Adding a code coverage badge to the README.

Notes:

  • Coverage will be recorded using Xdebug as the code coverage driver.
  • Recording code coverage can be slow and it is not needed to run it on each build to still get a good idea of the code coverage.
    With that in mind, code coverage is only recorded for high/low PHP.
  • The php-coveralls/php-coveralls package is used to convert the coverage information from a format as generated by PHPUnit to a format as can be consumed by Coveralls.
    • As this package is only needed for the upload to Coveralls, the package has not been added it to the composer.json file, but will be (global) installed in the GH Actions workflow only.
    • The php-coveralls/php-coveralls package is not 100% compatible with PHP 8.x (yet), so for uploading the coverage generated using PHP 8.x, we switch to PHP 7.4 for uploading the code coverage reports.
  • Coveralls requires a token to identify the repo and prevent unauthorized uploads.
    This token has been added to the repository secrets.
    People with admin access to the GH repo automatically also have access to the admin settings in Coveralls.
    If ever needed, the Coveralls token can be found (and regenerated) in the Coveralls admin for a repo.
    After regeneration, the token as stored in the GH repo Settings -> Secrets and Variables -> Actions -> Repository secrets should be updated.
  • As the workflow sends multiple code coverage reports to Coveralls, we need to tell it to process those reports in parallel and give each report a unique name.
    That's what the COVERALLS_PARALLEL and COVERALLS_FLAG_NAME` settings are about.
  • The coveralls-finish job will signal to Coveralls that all reports have been uploaded.
    This basically gives the "okay" to Coveralls to report on the changes in code coverage via a comment on a GitHub PR as well as via a status check.

The pertinent Coveralls settings used for this repo are:

  • "Only send aggregate Coverage updates to SCM": enabled
  • "Leave comments": enabled
  • "(Comment) Format": detailed
  • "Use Status API": enabled
  • "Coverage threshold for failure": 50%
  • "Coverage decrease threshold for failure": 1.0%

Test instructions

This PR can be tested by following these steps:

This commit makes the necessary changes to start recording and uploading code coverage data to Coveralls.

Includes:
* Updating the GH Actions `test` worflow to run the high/low PHP version builds with code coverage and upload the results.
* Updating the PHPUnit config to show an inline code coverage summary and provide the logging needed for Coveralls.
* Ignoring the potentially generated code coverage files in the `build/*` directory to prevent it being committed.
* Adding a script to the `composer.json` file to run code coverage.
* Adding a code coverage badge to the README.

Notes:
* Coverage will be recorded using `Xdebug` as the code coverage driver.
* Recording code coverage can be slow and it is not needed to run it on each build to still get a good idea of the code coverage.
    With that in mind, code coverage is only recorded for high/low PHP.
* The `php-coveralls/php-coveralls` package is used to convert the coverage information from a format as generated by PHPUnit to a format as can be consumed by Coveralls.
    - As this package is only needed for the upload to Coveralls, the package has **not** been added it to the `composer.json` file, but will be (global) installed in the GH Actions workflow only.
    - The `php-coveralls/php-coveralls` package is not 100% compatible with PHP 8.x (yet), so for uploading the coverage generated using PHP 8.x, we switch to PHP 7.4 for uploading the code coverage reports.
* Coveralls requires a token to identify the repo and prevent unauthorized uploads.
    This token has been added to the repository secrets.
    People with admin access to the GH repo automatically also have access to the admin settings in Coveralls.
    If ever needed, the Coveralls token can be found (and regenerated) in the Coveralls admin for a repo.
    After regeneration, the token as stored in the GH repo Settings -> Secrets and Variables -> Actions -> Repository secrets should be updated.
* As the workflow sends multiple code coverage reports to Coveralls, we need to tell it to process those reports in parallel and give each report a unique name.
    That's what the `COVERALLS_PARALLEL` and COVERALLS_FLAG_NAME` settings are about.
* The `coveralls-finish` job will signal to Coveralls that all reports have been uploaded.
    This basically gives the "okay" to Coveralls to report on the changes in code coverage via a comment on a GitHub PR as well as via a status check.

The pertinent Coveralls settings used for this repo are:
* "Only send aggregate Coverage updates to SCM": enabled
* "Leave comments": enabled
* "(Comment) Format": detailed
* "Use Status API": enabled
* "Coverage threshold for failure": 50%
* "Coverage decrease threshold for failure": 1.0%
@jrfnl jrfnl added yoast cs/qa changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog labels May 1, 2023
@jrfnl jrfnl added this to the 3.1 milestone May 1, 2023
@jrfnl jrfnl merged commit 44fee1c into develop May 1, 2023
@jrfnl jrfnl deleted the JRF/start-recording-code-coverage branch May 1, 2023 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog yoast cs/qa
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant