Document how to run coverage for repository idlelib files.

[terryjreedy]

BPO	30934
Nosy	@terryjreedy, @taleinat, @mlouielu, @csabella
PRs	bpo-30934: Document coverage details for idlelib tests #2711 [3.6] bpo-30934: Document coverage details for idlelib tests (GH-2711) #2726 bpo-30934: IDLE: Add Linux/MacOS coverage usage #2733 gh-75117: IDLE - add script for running coverage on tests #22694

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2017-07-15.01:19:45.246>
labels = ['expert-IDLE', 'type-feature', '3.8', '3.9', '3.10']
title = 'Document how to run coverage for repository idlelib files.'
updated_at = <Date 2020-10-14.14:18:48.433>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2020-10-14.14:18:48.433>
actor = 'taleinat'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2017-07-15.01:19:45.246>
creator = 'terry.reedy'
dependencies = []
files = []
hgrepos = []
issue_num = 30934
keywords = ['patch']
message_count = 12.0
messages = ['298379', '298384', '298386', '298412', '298415', '298416', '298436', '298446', '298494', '298569', '378580', '378619']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'taleinat', 'louielu', 'cheryl.sabella']
pr_nums = ['2711', '2726', '2733', '22694']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30934'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[terryjreedy]

I have been using coverage for a few years. Now that I am (again) not the only one writing IDLE tests (others did 3 and 4 years ago), I should document what I do and in particular the policy decision -- test with 'branch=True' -- and the IDLE-specific excludes:
.*# htest #
if not _utest:
if _htest:

[mlouielu]

After merging this one, I can provide the version of Linux and MacOS.

[terryjreedy]

That would be good; I might need to know someday;-).

[terryjreedy]

New changeset 95bebb7 by terryjreedy in branch 'master':
bpo-30934: Document coverage details for idlelib tests (bpo-2711)
95bebb7

[terryjreedy]

New changeset 65de1f3 by terryjreedy in branch '3.6':
[3.6] bpo-30934: Document coverage details for idlelib tests (GH-2711) (bpo-2726)
65de1f3

[terryjreedy]

I generalized the first part so only the .bat file needs a *nix version.

[mlouielu]

Upload the Linux/MacOS version of coverage usage.

[terryjreedy]

Cheryl, please try Louie's linux instructions and see if they work on Ubuntu.

[csabella]

I tried out the instructions. I guess my first question is how stand alone these instructions are and how much you expect someone to already know about coverage. What I mean is, I needed to go to the devguide to read more about coverage in order to run the steps here.

For example, the devguide suggests creating a venv before running pip install coverage. If I do that and go into the venv, then I could do (with python instead of ./python):

python -m coverage run --source Lib/idlelib Lib/idlelib/idle_test/test_config.py
python -m coverage report
python -m coverage html

Those steps seem to work.

Without the venv, ./python -m coverage doesn't work without the pip install (I get the 'No module named coverage' error). At some point (as per the devguide), I had also downloaded the in-development version, so I am able to run
./python ../coveragepy run --source Lib/idlelib Lib/idlelib/idle_test/test_config.py

At the time, the devguide suggested the in development version because it was more recent for 3.7 than the released version was, but it seems that the version on pip is the most current now.

Just like everything else, it seems that there are a few ways to go with this, but the venv with pip install seems to be the most foolproof. I don't know if you'd also want to include instructions for running coverage over all of idle_test instead of just one module.

(without the ./python)

[mlouielu]

I update how to prepare coverage on Linux and MacOS. I first thought user will prepare as Terry wrote at the top of 5. Test Coverage.

Cheryl's method is good, the update guide is borrow from here. First create a virtualenv, then install coverage via pip in venv. If user want to skip the step of venv, they can copy out the coverage package in venv/lib/ to Lib/ inside.

[terryjreedy]

Tal, this is an existing issue about documenting how to run IDLE coverage. It was left open to add something about the *nix equivalent of my Windows instructions. I would like to finish this, with updates elsewhere in the text.

I'm dubious about Louie's patch (PR 2733) for multiple reasons: it starts with venv's, which I consider a separate issue; it gives the commands that might go in a script, but does not comprise a script itself; I cannot tell if it uses an IDLE specific .coverage. Louie ceased IDLE involvement in Sept 2017, so I am closing his patch in favor of whatever you might contribute.

The .coveragerc copied into the README is missing some additions, and I plan to add some annotations to the original. Do you think it better to have a copy in README.txt, or a separate .coveragerc file in idle_test? Same question for cover.bat.

"5. Test Coverage" begins with "Install the coverage package into your Python 3.6 site-packages directory." That should be '3.x' or 'the most recent installed Python possible', so it is most likely to be compatible with current master branch python. I should also mention that we only need coverage on master, and have other revisions in mind.

Cheryl mentioned above instructions for coverage of all IDLE tests. I have the command, commented out, in my original cover.bat. After retesting, I could add it, commented out, to the copy.

[taleinat]

See a cross-platform script in PR #22694¹.

Footnotes

1. Incorrect PR number fixed by @erlend-aasland ↩