coverage 5.6b1 breaks support for testing code when installed

[FFY00]

Describe the bug
There are some cases where I want to test if my code works when installed. For this reason, I install the code and run the test suite. This has been working fine so far, but the latest beta breaks our setup.

cc @gaborbernat

To Reproduce

1. What version of Python are you using? 2.7, >=3.5
2. What version of coverage.py are you using? 5.6b1
3. What versions of what packages do you have installed?

apipkg==1.5,attrs==20.3.0,build @ file:///home/anubis/git/python-build/.tox/.tmp/package/1/build-0.3.1.post1.tar.gz,coverage==5.6b1,execnet==1.8.0,filelock==3.0.12,iniconfig==1.1.1,packaging==20.9,pep517==0.10.0,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.3,pytest-cov==2.11.1,pytest-forked==1.3.0,pytest-mock==3.5.1,pytest-xdist==2.2.1,toml==0.10.2

4. What code are you running? https://github.com/pypa/build, latest commit (bb8878c5fe0b13f6b1ef416dfd93c536ed01e625)
5. What commands did you run? tox -e py39 -e coverage

Patch to pin coverage:

diff --git a/setup.cfg b/setup.cfg
index f3831d6..9106151 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -53,6 +53,7 @@ test =
     filelock>=3
     pytest>=4
     pytest-cov>=2
+    coverage==5.6b1
     pytest-mock>=2
     pytest-xdist>=1.34
 typing =

Expected behavior
As explained above, I would like to be able to track the coverage of my code when I run the test suite with it installed.

[nedbat]

Thanks for the reproduction.

With coverage 5.5, this is measuring all of the code installed your virtualenv. Then in the reporting phase, you've used [paths] to map back to src, and -i to skip any file that doesn't exist. This results in reporting on just the src and tests directory, like you want.

With coverage 5.6b1, you will get the same results with [coverage.run] source = build, tests, and you can remove the -i switches. This will measure only the files you want.

I hope you'll agree that this is a better use of coverage.py. This was a good example of the kind of upgrading that people will need to do.

[FFY00]

Yes, this is a much better configuration IMO! Thanks 😊

[gaborbernat]

@nedbat in our case this works because we use a source layout. How would one measure coverage on installed files if they are not using the src layout?

Also note, while the suggested fix works, this makes 5.6b1 a breaking change for any CI doing any level of strictness check (which I'd hope are a lot). So perhaps this kind of major change should roll out as 6.0 instead.

[nedbat]

@gaborbernat you are probably right about the 6.0 version number.

If you weren't using the src layout, source_pkgs=build would work for you (and would also work for the src layout).

[nedbat]

I think this can be closed now... Please re-open if I am wrong.