New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
platform specific configuration #563
Comments
You can already get the effect of the regex being specified on the command line, because environment variables are expanded in the .coveragerc values. So you could use TOXENV in your coverage pragma, or some other environment variable you create just for that purpose. |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) I have env vars working. As platform specific code is one of the most common and unavoidable uses of I've build that type of thing into |
I'm not sure what that would look like, or how it would affect backward compatibility. |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) How about pseudo environment variables, using lower case to avoid conflicts with any env vars which might actually be set.
And if the env actually includes any of those keys, all are disabled...? |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) I creates two plugins which in concert solve this problem:
In action at coala/coala#4337 |
Help me understand how you use these plugins? Why do you need to reload the configuration settings? Aren't your different environments just different runs of coverage, with different config files? |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) https://github.com/coala/coala/blob/master/setup.cfg#L62
|
Original comment by Toshio Kuratomi (Bitbucket: toshio, GitHub: toshio) @jayvdb Do your additions allow excluding coverage reports for just python major version? (Only report on Python2 or only report on Python3)? PYTHON_VERSION looks like it will expand to the full MAJOR.MINOR.MICRO version which seems less helpful. (you needing to specify every Python version that the tests will run on. If the config is to be generic for any random OSS contributor to the project, that quickly gets out of hand. |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) No, we use it for minor versions. It is a bit awkward. However PYTHON_VERSION expands to only MAJOR.MINOR , so .MICRO isnt involved. I see https://www.python.org/dev/peps/pep-0496/#examples gives a bad example with .MICRO, which is useless and actually will break some versions of setuptools/pip/etc. Here is code which only needs coverage on Python 3.4 , and not 3.5/3.6 If we do builds on Python 3.7, we would need to add 3.7 there. |
@jayvdb Could you define these environment variables in your actual environment, so they are available to the config parser? It could be done with a Python program that sets the environment and then runs another command (like "coverage"), so that you didn't have to fiddle around in the coverage internals. Then you can define whatever set of environment variables you like. |
Original comment by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb) of course. We were doing that before coala/coala#4337 . Im looking for coveragepy to have a builtin solution for platform specific code. I did a prototype using plugins, and it is a bit hacky due to how the coverage config is loaded. If the concept is accepted, a patch to core can improve on the prototypes in many ways. |
My concern about this path is that the set of conditions is going to vary from project to project. Is 3.6.2 a different platform than 3.6.3? Is Ubuntu different than Arch? Is Ubuntu different than Debian? I would rather have the code that makes those determinations be outside of coverage.py. One idea (mentioned above) is a program runner that sets the environment variables. Another would be to add a feature to coverage.py so that it reads a specific env file to set variables in addition to reading them from the environment. Then your program could write that file before running coverage. Though again, the existing mechanism (the environment) is specifically designed for this. We shouldn't need to invent new mechanisms to get this to work. I understand what you mean about wanting the solution in a common place. Your program that makes these determinations can become a tool used by many projects if they need this kind of fine-slicing behavior. |
Original comment by Barry Warsaw (Bitbucket: warsaw, GitHub: warsaw) Here's another case that I've had to deal with recently: https://github.com/brettcannon/importlib_resources/blob/master/importlib_resources/_py3.py#L18 In this case I need to specify coverage for ranges of Python 3 minor versions. Through a terrible combination of my tox.ini and coverage.ini files, I've managed to make it work, but I don't like it. It's hard to reason about when reading the code, and it isn't very robust. This may be off-topic for this particular issue, but having minor version ranges would be very helpful. |
Original comment by Toshio Kuratomi (Bitbucket: toshio, GitHub: toshio) Looking at Barry's case, I think we do need some additional support in coverage for a clean solution. Right now coverage gives us a way to specify in coverage that a pattern in the code should be excluded. What Barry is looking at is needing to set a pattern in code (a range of Python versions). My initial thought is that that would need for coverage to have a pattern that it extracts from the code (like '#pragma: .*$') and then coverage would have to parse that to determine what to do. We can't do that with a purely external tool but perhaps coverage could give us hook points to do it in a plugin? |
Barry's case just convinces me more and more that the actual conditions needed will vary wildly from project to project. I really don't want coverage.py to be dictating what special conditions you can select for in pragmas. Using environment variables in .coveragerc has already been more powerful than anything I could have built in coverage.py directly. What I'm wondering is how to add something akin to environment variables in .coveragerc, to provide a more useful configuration parameterization mechanism. Then you can use your own program to determine the pragmas to put into effect, using whatever logic you want. For example, what if the .coveragerc could name another file to read as part of the configuration? Then your prelude program could write a bunch of A new plugin that could modify the configuration would also do the trick. |
Original comment by Toshio Kuratomi (Bitbucket: toshio, GitHub: toshio) None of that would help for Barry's case. A configuration plugin would help for many other cases (including mine), though. In the non-Barry cases, people are specifying something in our code which can be exactly matched by something that can be provided to coverage. If I put (Note: it would help in writing plugins or other tools that deal with this if the environment variables were expanded when the exclude is used. Right now, they are expanded when the configuration is read in which means that tools have to set the environment variables very early in the process. For instance, I wrote a pytest plugin to address this and had to make it a hookwrapper in order to run before the pytest_cov plugin which read the coverage configuration at startup.) In Barry's case, though, he wants to specify a range like this in his code:
The problem with coverage's current exclude configuration is that it has no way to specify that I think the idea of reading part of the configuration from another file largely misses the point. People could write multiple .coveragerc's already. The problem is that selecting which one to use on any particular test run then gets messy. People want something that they can set up once and then just let it do its thing automatically. An external tool can handle some cases but it has the drawback of having to fuss with it every time you set up your testing in something new (tox vs travis vs appveyor vs raw pytest vs [....]). An external plugin to coverage (one which enables the desired behaviour by virtue of being pip installed on the system/virtualenv) seems like the best bet which doesn't put the code directly into coverage itself. But to do that, it needs to have access to hook into coverage's work flow at the appropriate time. |
OK, here's the plan I have in mind: we'll have a new kind of plugin:
The plugin would have a chance to use Sound good? |
I implemented this in 781b3c282758 (bb). Try it out, let me know what you think. |
This was included in coverage.py v4.5, released today. |
There's a new coverage plugin that can do these sorts of conditionals: https://sobolevn.me/2020/02/conditional-coverage |
Originally reported by John Vandenberg (Bitbucket: jayvdb, GitHub: jayvdb)
If a block has two platform/OS-specific branches, currently the only way to reach 100% with a single configuration is to use
#pragma: no cover
on both branches, ignoring the block when coverage is run on any platform/OS.It is useful for each platform/OS to require coverage for their own branch.
This can be currently be achieved by using a different configuration file for each platformOS.
It would be helpful that the configuration allows for different ignore patterns for platform/OS, so only one configuration file is needed for all platforms.
One easy way to do this is to iterate the section names in the ini file, extracting any section with
pytest
in the name, and then finding the most appropriate section for the host platform/OS.Another approach is to allow the ignore regex to be specified on the command line.
The text was updated successfully, but these errors were encountered: