Test faults

[brownbaerchen]

I add soft faults via a hook and I used to do it with inheritance. However, this was not compatible with the current hook API anymore.
I used the opportunity to clean up most of the hooks in the resilience project folder and adapted the notebooks to the new changes.

Also, I added some tests for generating fault statistics. These are part of the mpi tests because different processors generate statistics for different strategies in parallel. All individual experiments are run in serial.
The tests are a bit flaky because I get overflow errors on GitHub, but not on my laptop. The result is a test that is not as strict as I would like, but I think it's better than nothing. Unfortunately I did not find a satisfactory solution to fix the overflow behaviour across machines so far.

Since I need to generate plots with the fault stuff, I wanted to get everything working again and maybe come back to the flaky test when I have more experience/patience/wisdom/beer/time.

[codecov]

Codecov Report

Base: 68.66% // Head: 70.59% // Increases project coverage by +1.93% 🎉

Coverage data is based on head (68f3111) compared to base (8b987cd).
Patch coverage: 65.74% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
+ Coverage   68.66%   70.59%   +1.93%     
==========================================
  Files         229      229              
  Lines       19643    19646       +3     
==========================================
+ Hits        13487    13869     +382     
+ Misses       6156     5777     -379     

Impacted Files	Coverage Δ
pySDC/projects/Resilience/heat.py	0.00% <0.00%> (ø)
pySDC/projects/Resilience/advection.py	75.47% <33.33%> (+3.05%)	⬆️
pySDC/projects/Resilience/piline.py	84.93% <33.33%> (-1.18%)	⬇️
pySDC/projects/Resilience/hook.py	70.00% <40.00%> (-30.00%)	⬇️
pySDC/projects/Resilience/fault_stats.py	43.27% <50.00%> (+43.27%)	⬆️
pySDC/implementations/hooks/log_errors.py	86.20% <66.66%> (-13.80%)	⬇️
pySDC/projects/Resilience/vdp.py	93.92% <85.71%> (+1.74%)	⬆️
pySDC/implementations/hooks/log_solution.py	100.00% <100.00%> (ø)
pySDC/projects/Resilience/Lorenz.py	84.50% <100.00%> (ø)
pySDC/projects/Resilience/accuracy_check.py	97.18% <100.00%> (-0.10%)	⬇️
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[brownbaerchen]

I am confused how adding more tests reduced the coverage this much... Do files that don't get tested at all not count into the percentage? Otherwise it is weird to me that testing half of the fault_stats file is worse than not testing it at all. The other changes in coverage are rather small...

[pancetta]

I am confused how adding more tests reduced the coverage this much... Do files that don't get tested at all not count into the percentage? Otherwise it is weird to me that testing half of the fault_stats file is worse than not testing it at all. The other changes in coverage are rather small...

Don't know. Untested files are of course counted toward the overall coverage (unless they are in the test or playground directories, which are explicitly excluded from coverage. You can look at the coverages of both master branch and your PR here. Does that help?

The "classical" coverage report (see artifacts of your CI run) show the same numbers, btw.

[tlunet]

Seems like adding custom-made classes for a given project (here hook) without specific testing is a coverage killer 😅

[brownbaerchen]

In the coverage report only files show up that are tested to some extent and at the bottom it says "33 empty files skipped."
I am actually pretty sure that files which are not tested don't count towards the overall coverage. After all, when running

coverage run -m pytest --continue-on-collection-errors -v --durations=0 pySDC/tests -m ${{ matrix.env }}

only the tests directory is tested, right? Why would coverage check other directories for tests as well?
When I run only

coverage run -m pytest --continue-on-collection-errors -v --durations=0 pySDC/tests/test_projects/test_resilience/test_fault_injection.py -m mpi4py

to execute only one of the tests which I added, I get a coverage of 51%, but there is no way that I am execution 51% of the code with this. Just 51% of the files which participate in this test are executed. Right?
Maybe I am missing something. These toml files and so an are still quite opaque to me...

[brownbaerchen]

I failed at the coverage thing :( Still convinced that this PR improves coverage, though

[pancetta]

Please merge latest commits from master.. happy now? 😢

[pancetta]

What's the status here?

[brownbaerchen]

Codecov is complaining that the test coverage is too low in the files that I touched. Given that the overall test coverage is still up by 2%, I suggest merging this anyways.
I resolved some merge conflicts, let's see if the tests go through before doing anything.