Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test improvements, move to Pytest #227

merged 5 commits into from Feb 25, 2018

Test improvements, move to Pytest #227

merged 5 commits into from Feb 25, 2018


Copy link

@latk latk commented Feb 22, 2018

This PR includes a bunch of testing improvements. Importantly, this changes the testrunner from Nose to Pytest, which allows us to simplify the tests in some places.

Why Pytest?

  • it is maintained
  • it makes tests really easy – just use "assert"
  • it produces really helpful error reports
  • it auto-captures the STDOUT of a test, including subprocesses
  • it can automatically gather coverage for subprocesses
  • super simple test selection, without having to run the whole test suite
  • it has a rich plugin ecosystem
  • compare also: Switching from nose to py.test at Mozilla

This means that in most cases, the test assertions are no longer provided by Pyutilib. With one exception: the Pyutilib XML comparison has no equivalent, except perhaps in lxml (which we are not currently using). The Pyutilib diffs for file comparisons are better than the Pytest string comparison diffs, but not by so much that I want to dig through the Pyutilib source code to find out how to use them.

Assorted other changes:

  • In, replace our StringIOBuffered with Pytest's capsys fixture.
  • The documentation example tests are now executed on Appveyor and Travis.
  • The --html-details reports are now checked if a corresponding reference file exists. So far, only some newer tests include these files.
  • Some whitespace adjustments in the HTML templates.
  • The html or cgi imports were removed, as their escape() functions differ in behaviour. That makes tests difficult. Instead, I added a minimal escaping function that only escapes < and &, i.e. is suitable for escaping text contents but not attributes in HTML.
latk added 5 commits Feb 22, 2018
Pytest has a number of advantages over Nose:

 - it is maintained
 - it produces really helpful error reports
 - it makes tests really easy – just use "assert"
 - it auto-captures the STDOUT of a test, including subprocesses
 - it can automatically cover subprocesses
 - it has a rich plugin ecosystem
Pytest provides a built-in mechanism for capturing STDOUT by using the
"capsys" dependency.
The test suite will now verify the --html-details reports if the reports
are stored as a reference file. This led to the following changes:

  - Existing source reports were updated to hold a full timestamp (not
    just date).
  - All trailing whitespace was stripped from the HTML reports.
  - This required adjustements to the HTML templates.
  - The "cgi.escape" and "html.escape" functions differ slightly. To
    simplify tests, a custom "html_escape()" function was introduced.
  - To make test failures more debuggable, a HTML comparison failure
    will specify the files being compared.
Copy link

@codecov codecov bot commented Feb 22, 2018

Codecov Report

Merging #227 into master will decrease coverage by 0.16%.
The diff coverage is 98.4%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #227      +/-   ##
- Coverage   84.17%    84%   -0.17%     
  Files          11     11              
  Lines        1251   1194      -57     
  Branches      248    248              
- Hits         1053   1003      -50     
+ Misses        138    131       -7     
  Partials       60     60
Impacted Files Coverage Δ
gcovr/ 94.04% <100%> (ø) ⬆️
gcovr/tests/ 100% <100%> (ø) ⬆️
gcovr/tests/ 97.56% <97.22%> (-2.44%) ⬇️
gcovr/ 95.62% <0%> (+6.56%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b0e68f...12e0ec0. Read the comment docs.

mayeut approved these changes Feb 25, 2018
@latk latk merged commit 12e0ec0 into gcovr:master Feb 25, 2018
4 checks passed
@latk latk deleted the test-improvements branch Feb 25, 2018
@latk latk removed the needs review label Feb 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants