Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/QA: start recording code coverage #126

Merged
merged 1 commit into from
May 1, 2023
Merged

Conversation

jrfnl
Copy link
Contributor

@jrfnl jrfnl commented May 1, 2023

This commit makes the necessary changes to start recording and uploading code coverage data to Coveralls.

Includes:

  • Updating the GH Actions test worflow to run the high/low PHP version builds with code coverage and upload the results.
  • Updating the PHPUnit config to show an inline code coverage summary, improve the filter configuration and force the use of @covers tags.
  • Ignoring the potentially generated code coverage files in the build/* directory to prevent it being committed.
  • Adding a script to the composer.json file to run code coverage.
  • Adding a code coverage badge to the README.

Notes:

  • Coverage will be recorded using Xdebug as the code coverage driver.
  • Recording code coverage can be slow and it is not needed to run it on each build to still get a good idea of the code coverage. With that in mind, code coverage is only recorded for high/medium/low PHP.
  • The php-coveralls/php-coveralls package is used to convert the coverage information from a format as generated by PHPUnit to a format as can be consumed by Coveralls. - As this package is only needed for the upload to Coveralls, the package has not been added it to the composer.json file, but will be (global) installed in the GH Actions workflow only. - The php-coveralls/php-coveralls package is not 100% compatible with PHP 8.x (yet), so for uploading the coverage generated using PHP 8.x, we switch to PHP 7.4 for uploading the code coverage reports. - For uploading code coverage with PHP < 5.5, php-coveralls/php-coveralls 1.x is needed, but that version of the package doesn't play nice with GH Actions. To get round that, the job which runs on PHP < 5.5, will switch to PHP 7.4 for uploading the code coverage reports.
  • Coveralls requires a token to identify the repo and prevent unauthorized uploads. This token has been added to the repository secrets. People with admin access to the GH repo automatically also have access to the admin settings in Coveralls. If ever needed, the Coveralls token can be found (and regenerated) in the Coveralls admin for a repo. After regeneration, the token as stored in the GH repo Settings -> Secrets and Variables -> Actions -> Repository secrets should be updated.
  • As the workflow sends multiple code coverage reports to Coveralls, we need to tell it to process those reports in parallel and give each report a unique name. That's what the COVERALLS_PARALLEL and COVERALLS_FLAG_NAME` settings are about.
  • The coveralls-finish job will signal to Coveralls that all reports have been uploaded. This basically gives the "okay" to Coveralls to report on the changes in code coverage via a comment on a GitHub PR as well as via a status check.

The pertinent Coveralls settings used for this repo are:

  • "Only send aggregate Coverage updates to SCM": enabled
  • "Leave comments": enabled
  • "(Comment) Format": detailed
  • "Use Status API": enabled
  • "Coverage threshold for failure": 20%
  • "Coverage decrease threshold for failure": 1.0%

@jrfnl jrfnl added this to the 1.x Next Release milestone May 1, 2023
@jrfnl jrfnl force-pushed the JRF/improve-code-coverage-setup branch from bb0f8c7 to 30a3fbc Compare May 1, 2023 20:25
This commit makes the necessary changes to start recording and uploading code coverage data to Coveralls.

Includes:
* Updating the GH Actions `test` worflow to run the high/low PHP version builds with code coverage and upload the results.
* Updating the PHPUnit config to show an inline code coverage summary, improve the `filter` configuration and force the use of `@covers` tags.
* Ignoring the potentially generated code coverage files in the `build/*` directory to prevent it being committed.
* Adding a script to the `composer.json` file to run code coverage.
* Adding a code coverage badge to the README.

Notes:
* Coverage will be recorded using `Xdebug` as the code coverage driver.
* Recording code coverage can be slow and it is not needed to run it on each build to still get a good idea of the code coverage.
    With that in mind, code coverage is only recorded for high/medium/low PHP.
* The `php-coveralls/php-coveralls` package is used to convert the coverage information from a format as generated by PHPUnit to a format as can be consumed by Coveralls.
    - As this package is only needed for the upload to Coveralls, the package has **not** been added it to the `composer.json` file, but will be (global) installed in the GH Actions workflow only.
    - The `php-coveralls/php-coveralls` package is not 100% compatible with PHP 8.x (yet), so for uploading the coverage generated using PHP 8.x, we switch to PHP 7.4 for uploading the code coverage reports.
    - For uploading code coverage with PHP < 5.5, `php-coveralls/php-coveralls` `1.x` is needed, but that version of the package doesn't play nice with GH Actions. To get round that, the job which runs on PHP < 5.5, will switch to PHP 7.4 for uploading the code coverage reports.
* Coveralls requires a token to identify the repo and prevent unauthorized uploads.
    This token has been added to the repository secrets.
    People with admin access to the GH repo automatically also have access to the admin settings in Coveralls.
    If ever needed, the Coveralls token can be found (and regenerated) in the Coveralls admin for a repo.
    After regeneration, the token as stored in the GH repo Settings -> Secrets and Variables -> Actions -> Repository secrets should be updated.
* As the workflow sends multiple code coverage reports to Coveralls, we need to tell it to process those reports in parallel and give each report a unique name.
    That's what the `COVERALLS_PARALLEL` and COVERALLS_FLAG_NAME` settings are about.
* The `coveralls-finish` job will signal to Coveralls that all reports have been uploaded.
    This basically gives the "okay" to Coveralls to report on the changes in code coverage via a comment on a GitHub PR as well as via a status check.

The pertinent Coveralls settings used for this repo are:
* "Only send aggregate Coverage updates to SCM": enabled
* "Leave comments": enabled
* "(Comment) Format": detailed
* "Use Status API": enabled
* "Coverage threshold for failure": 20%
* "Coverage decrease threshold for failure": 1.0%
@jrfnl jrfnl force-pushed the JRF/improve-code-coverage-setup branch from 30a3fbc to 0d65a18 Compare May 1, 2023 20:26
@jrfnl jrfnl merged commit 51ff860 into main May 1, 2023
@jrfnl jrfnl deleted the JRF/improve-code-coverage-setup branch May 1, 2023 20:28
@enricobattocchi enricobattocchi modified the milestones: 1.x Next Release, 2.0 Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants