Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test improvements, move to Pytest #227

Merged
merged 5 commits into from Feb 25, 2018

Conversation

Projects
None yet
2 participants
@latk
Copy link
Member

commented Feb 22, 2018

This PR includes a bunch of testing improvements. Importantly, this changes the testrunner from Nose to Pytest, which allows us to simplify the tests in some places.

Why Pytest?

  • it is maintained
  • it makes tests really easy – just use "assert"
  • it produces really helpful error reports
  • it auto-captures the STDOUT of a test, including subprocesses
  • it can automatically gather coverage for subprocesses
  • super simple test selection, without having to run the whole test suite
  • it has a rich plugin ecosystem
  • compare also: Switching from nose to py.test at Mozilla https://agopian.info/presentations/2015_06_djangocon_europe/

This means that in most cases, the test assertions are no longer provided by Pyutilib. With one exception: the Pyutilib XML comparison has no equivalent, except perhaps in lxml (which we are not currently using). The Pyutilib diffs for file comparisons are better than the Pytest string comparison diffs, but not by so much that I want to dig through the Pyutilib source code to find out how to use them.

Assorted other changes:

  • In test_args.py, replace our StringIOBuffered with Pytest's capsys fixture.
  • The documentation example tests are now executed on Appveyor and Travis.
  • The --html-details reports are now checked if a corresponding reference file exists. So far, only some newer tests include these files.
  • Some whitespace adjustments in the HTML templates.
  • The html or cgi imports were removed, as their escape() functions differ in behaviour. That makes tests difficult. Instead, I added a minimal escaping function that only escapes < and &, i.e. is suitable for escaping text contents but not attributes in HTML.

latk added some commits Feb 22, 2018

switch testrunner from nose to pytest
Pytest has a number of advantages over Nose:

 - it is maintained
 - it produces really helpful error reports
 - it makes tests really easy – just use "assert"
 - it auto-captures the STDOUT of a test, including subprocesses
 - it can automatically cover subprocesses
 - it has a rich plugin ecosystem
simplify test_args using pytest
Pytest provides a built-in mechanism for capturing STDOUT by using the
"capsys" dependency.
add tests for --html-details
The test suite will now verify the --html-details reports if the reports
are stored as a reference file. This led to the following changes:

  - Existing source reports were updated to hold a full timestamp (not
    just date).
  - All trailing whitespace was stripped from the HTML reports.
  - This required adjustements to the HTML templates.
  - The "cgi.escape" and "html.escape" functions differ slightly. To
    simplify tests, a custom "html_escape()" function was introduced.
  - To make test failures more debuggable, a HTML comparison failure
    will specify the files being compared.
@codecov

This comment has been minimized.

Copy link

commented Feb 22, 2018

Codecov Report

Merging #227 into master will decrease coverage by 0.16%.
The diff coverage is 98.4%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #227      +/-   ##
========================================
- Coverage   84.17%    84%   -0.17%     
========================================
  Files          11     11              
  Lines        1251   1194      -57     
  Branches      248    248              
========================================
- Hits         1053   1003      -50     
+ Misses        138    131       -7     
  Partials       60     60
Impacted Files Coverage Δ
gcovr/html_generator.py 94.04% <100%> (ø) ⬆️
gcovr/tests/test_args.py 100% <100%> (ø) ⬆️
gcovr/tests/test_gcovr.py 97.56% <97.22%> (-2.44%) ⬇️
gcovr/cobertura_xml_generator.py 95.62% <0%> (+6.56%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b0e68f...12e0ec0. Read the comment docs.

@mayeut

mayeut approved these changes Feb 25, 2018

@latk latk merged commit 12e0ec0 into gcovr:master Feb 25, 2018

4 checks passed

codecov/patch 98.4% of diff hit (target 84.17%)
Details
codecov/project Absolute coverage decreased by -0.16% but relative coverage increased by +14.22% compared to 5b0e68f
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@latk latk deleted the latk:test-improvements branch Feb 25, 2018

@latk latk removed the needs review label Feb 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.