Tweak coverage

[dopplershift]

Mainly to turn on coverage for all testing on AppVeyor, as well as tighten our expected threshold for the test lines.

WIP because I want to investigate some of the test files that aren't at 100%.

[dopplershift]

Oh goody, we had two tests with the same name, so only the second ran (luckily it now succeeds here, and it's only a smoke test).

Anyone know a good way to check for this within our test suite--besides checking files with seemingly low test coverage %?

[dopplershift]

More fun:

• Completely duplicated test methods in Test_detrend in test_mlab.py
• In test_pickle.py, recursive_pickle (and hence depth_getter) are completely unused in living code--they're commented out debugging helpers. Suggestions on where else we could put them? Or can we remove?

[QuLogic]

For top-level functions, this is relatively easy:

$ grep -IRo '^def[[:space:]]\+[A-Za-z0-9_]\+' | sort | uniq -d
test_axes.py:def test_pcolorargs
test_contour.py:def test_contour_manual_labels

For class methods, maybe a short Python script:

import sys
from collections import Counter

for name in sys.argv[1:]:
    methods = Counter()
    class_name = None
    with open(name, 'r') as f:
        for line in f:
           if line.startswith('class'):
               class_name = line.split()[1].split('(')[0].strip()
           elif line.startswith('def'):
               class_name = None
           elif class_name and line.startswith('    def'):
               method_name = line.split()[1].split('(')[0].strip()
               methods[class_name + '.' + method_name] += 1
    duplicates = [name for name, count in methods.most_common() if count != 1]
    if duplicates:
        print(name, duplicates)

test_mlab.py ['Test_detrend.test_detrend_mean_2D_axism1', 'Test_detrend.test_detrend_mean_2D_none', 'Test_detrend.test_detrend_mean_2D_axis0', 'Test_detrend.test_detrend_str_linear_2d_slope_off_axis0', 'Test_detrend.test_detrend_mean_2D_none_T', 'Test_detrend.test_detrend_mean_2D_axis1', 'Test_detrend.test_detrend_detrend_linear_1d_slope_off_axis1']
test_skew.py ['SkewXTick.tick2On', 'SkewXTick.gridOn', 'SkewXTick.tick1On', 'SkewXTick.label2On', 'SkewXTick.label1On']

[QuLogic]

For top-level classes, too:

$ grep -IRo '^class[[:space:]]\+[A-Za-z0-9_]\+' | sort | uniq -d
test_ticker.py:class TestLogFormatter

[tacaswell]

I am 👍 on removing the pickle debugging helpers. I have 0 pity on people depending on helper functions from in our tests (not in testing, in the tests). We have already broken those folks by default by not including the tests anyway.

[dstansby]

Travis is failing on this image comparison. I'm guessing this just needs an updated image, since it's been a while since this test was ever run.

[dopplershift]

Updated. Surprisingly, that's the only test that had that problem. The other newly activated tests pass, thankfully.

[phobson]

How are we handling regenerating images as a policy?

See #7905

I think we'll shortly have to revert this changed image when/if #7970 is merged.

[dopplershift]

I'm happy to wait on this until the other is decided upon.

[phobson]

Cool. I think everything else is A+, 👍 so far.

[dopplershift]

It's been a good exercise. All of the test files with significant uncovered lines were caused by either dead code or tests that weren't running.

I noticed a significant number of uncovered lines left are caused by pass. Thoughts on adding pass to our exclude list in .coveragerc? Or is that cheating?

[dopplershift]

Another question: Currently, our matplotlib.testing stands at ~57% coverage because all the nose stuff is disabled. Do we want to exclude the nose testing code from the report?

[NelleV]

I think both solutions are fine, with a slight preference in keeping it.
The day we will get rid of this code, our coverage will just jump up.

[tacaswell]

do we need the second part of the assert with pytest?

[dopplershift]

Oh, I guess not.

[tacaswell]

These are slightly different tests as assert 'aardvark' passes, but assert 'aardvark' == True does not.

Do we care about this change?

[dopplershift]

Anyone have opinions on whether we should be ignoring pass lines for test coverage? We currently exclude:

    raise NotImplemented
    def __str__
    def __repr__
    if __name__ == .__main__.:

[tacaswell]

I don't think we should skip pass lines. Either we have non-functional code (as in the body is just pass and it really should be NotImplemented or they pass lines are in fall-troughs of input sanitation/validation which if we miss them, there is a case of inputs we are missing in the tests.

[dopplershift]

I think a lot of these are stubs in interfaces in the tests. I'm leaning towards turning them into some variety of NotImplemented such that they break should they be used.

[dstansby]

Looks good.

re. the new test image, the image would/should have been regenerated a while ago, so adding it even if it'll be changed soon seems fine to me.