Update default pytest output

[ischoegl]

Changes proposed in this pull request

• Switch pytest output to verbose (-v option)
• Hide information on longest runs by default
• Implement boolean SCons variable show_long_tests

The rationale for switching to verbose output is that segfaults are difficult to locate with the progress bar. Further, the longest running tests can still be displayed using

scons test-python-reaction show_long_tests=yes

While test_convert.py (scons test-python-convert) creates numerous WARNING messages from captured log calls - which may confuse some users - I am leaving those for another PR as those may or may not be a feature.

If applicable, fill in the issue number this pull request is fixing

Fixes #1026

Checklist

• There is a clear use-case for this code change
• The commit message has a short title & references relevant issues
• Build passes (scons build & scons test) and unit tests address code coverage
• Style & formatting of contributed code follows contributing guidelines
• The pull request is ready for review

[bryanwweber]

Thanks @ischoegl This looks good to me. Just one small question.

[bryanwweber]

I'll wait for @speth to review, since he was specifically requested.

[codecov]

Codecov Report

Merging #1034 (03c1f33) into main (19f1593) will increase coverage by 0.38%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1034      +/-   ##
==========================================
+ Coverage   72.54%   72.92%   +0.38%     
==========================================
  Files         355      357       +2     
  Lines       46422    47120     +698     
==========================================
+ Hits        33675    34363     +688     
- Misses      12747    12757      +10     

Impacted Files	Coverage Δ
src/kinetics/GasKinetics.cpp	93.03% <0.00%> (-4.41%)	⬇️
src/kinetics/ReactionRate.cpp	96.00% <0.00%> (-4.00%)	⬇️
src/kinetics/Kinetics.cpp	83.93% <0.00%> (-2.54%)	⬇️
include/cantera/kinetics/RxnRates.h	91.91% <0.00%> (-0.06%)	⬇️
src/base/ctexceptions.cpp	85.00% <0.00%> (ø)
src/kinetics/ReactionData.cpp	100.00% <0.00%> (ø)
include/cantera/kinetics/Reaction.h	100.00% <0.00%> (ø)
include/cantera/kinetics/GasKinetics.h	100.00% <0.00%> (ø)
test/transport/transportPolynomials.cpp	100.00% <0.00%> (ø)
test/kinetics/kineticsFromScratch3.cpp	100.00% <0.00%> (ø)
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19f1593...03c1f33. Read the comment docs.

[ischoegl]

@bryanwweber and @speth ... I updated the interface so the verbose output is optional, e.g.

scons test-python-reaction verbose_tests=yes

So, the changes are as follows:

• verbose output is not shown by default (keeping existing behavior), but can be turned on using verbose_tests=yes
• long tests are not shown by default (changing existing behavior), but can be turned on using show_long_tests=yes

One caveat is that verbosity only affects pytest; I found some external work on googletest verbosity, but that goes well beyond the scope of what I want to do here as I'm not familiar with the setup of these runners.

[ischoegl]

Introducing the verbose_tests flag has the advantage that it allows for disabling logger WARNING displays by default, which may be confusing to many especially for test_python_convert.py. The default display no longer shows warnings, I.e.:

docker@xps13:/src$ scons test-python-convert                 
[...]

* INFO: using Cantera module found at this location:
*      '/src/build/python/cantera/__init__.py'
* INFO: Cantera version: 2.6.0a2
* INFO: Git commit: b8a692300 

============================= test session starts =============================
platform linux -- Python 3.6.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /src
collected 141 items                                                           

../../build/python/cantera/test/test_convert.py ....................... [ 16%]
....................................................................... [ 66%]
...............................................                         [100%]

=================================== PASSES ====================================
---------------- generated xml file: /src/test/work/pytest.xml ----------------
============================ 141 passed in 46.47s =============================
scons: done building targets.
docker@xps13:/src$ 

The WARNING messages will be displayed if verbose_tests=yes.

[speth]

I think the implementation of the new options here is fine.

Given that the initial purpose of printing out test durations was as a diagnostic for the CI runs, I think both show_long_test and verbose_tests should be set to y for all CI runs.

[ischoegl]

@speth ... thank you for your feedback! Since I initially looked at this, I have started to to believe that there is a benefit to having shorter output by default for users that compile Cantera from source. For development, it is a different issue, where we often need more detailed information, but the flags are easy to toggle.

Given that the initial purpose of printing out test durations was as a diagnostic for the CI runs, I think both show_long_tests and verbose_tests should be set to y for all CI runs.

As an alternative to changing the default (where suppressing detailed output imho makes sense), I just flipped the flags for GH actions work flows. What do you think?

[speth]

As an alternative to changing the default (where suppressing detailed output imho makes sense), I just flipped the flags for GH actions work flows. What do you think?

Yes, that's exactly what I was suggesting. Sorry if that wasn't clear. With this update, I think this is good.