"coverage" of Python regrtest cannot see initial import of libs

[brandon-rhodes]

BPO	11561
Nosy	@Yhg1s, @brettcannon, @ncoghlan, @nedbat, @ezio-melotti, @merwok, @sandrotosi, @ericsnowcurrently
Files	fullcoverage.patch fullcoverage2.patch: Second version of a full-coverage tool for Python's own test suite stdlib_landmark.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2011-09-06.17:50:07.137>
created_at = <Date 2011-03-15.22:09:10.427>
labels = []
title = '"coverage" of Python regrtest cannot see initial import of libs'
updated_at = <Date 2011-09-06.17:50:07.136>
user = 'https://bugs.python.org/brandon-rhodes'

bugs.python.org fields:

activity = <Date 2011-09-06.17:50:07.136>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2011-09-06.17:50:07.137>
closer = 'brett.cannon'
components = ['Demos and Tools']
creation = <Date 2011-03-15.22:09:10.427>
creator = 'brandon-rhodes'
dependencies = []
files = ['21232', '22823', '23107']
hgrepos = []
issue_num = 11561
keywords = ['patch']
message_count = 13.0
messages = ['131051', '131053', '140135', '140164', '141529', '141577', '142657', '142659', '143582', '143584', '143603', '143610', '143635']
nosy_count = 9.0
nosy_names = ['twouters', 'brett.cannon', 'ncoghlan', 'nedbat', 'ezio.melotti', 'eric.araujo', 'sandro.tosi', 'brandon-rhodes', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'patch review'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue11561'
versions = ['Python 3.3']

[brandon-rhodes]

When running the Python regression tests in "coverage", the initial outer level of interpreted code in several standard library modules shows as not having been covered by the tests, because they were imported during the Python boot process and were already loaded when the "coverage" command got control.

[brandon-rhodes]

Here is a module that solves this problem if the tests are run with the "fullcoverage" directory at the front of the PYTHONPATH, like this:

PYTHONPATH=Tools/fullcoverage ./python -m coverage run --pylib Lib/test/regrtest.py test_copy

[merwok]

Thanks for your work on this. I made some minor comments on Rietveld (you should get an email), waiting for the real import experts to comment on the meat of the patch.

[brettcannon]

Which I have not forgotten about. Just waiting until I have the time to get
to this.

On Mon, Jul 11, 2011 at 08:50, Éric Araujo <report@bugs.python.org> wrote:

Éric Araujo <merwok@netwok.org> added the comment:

Thanks for your work on this. I made some minor comments on Rietveld (you
should get an email), waiting for the real import experts to comment on the
meat of the patch.

----------
nosy: +eric.araujo

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11561\>

---

[brandon-rhodes]

Éric, I think your suggestions are all good ones, and I have incorporated them into the file. (But do note that the departures we are now making from Ned's own copy of the tracer code ­— removing the commented-out debugging statement, and the long comment, and the inheritance from object — might make it harder to bring in changes from his own copy if he should ever further improve it.) I have tried to write the comments to be more informative, while also addressing your own ideas; let me know if you like the result!

Oh: and, I am continuing to use this new file in my own work on the Python core, and it has been working fine — so no problems with the actual code have developed over these first 3+ months of use.

[merwok]

I think your suggestions are all good ones, and I have incorporated
them into the file.
Great :) You left some commented-out debugging prints.

(But do note that the departures we are now making from Ned's own
copy of the tracer code ­— removing the commented-out debugging
statement, and the long comment, and the inheritance from object —
might make it harder to bring in changes from his own copy if he
should ever further improve it.)
I’m not concerned by this. I’ve been doing merges all the time for more that a year, and having a few diverging lines is not a big deal.

I have tried to write the comments to be more informative, while also
addressing your own ideas; let me know if you like the result!
I do!

Oh: and, I am continuing to use this new file in my own work on the
Python core, and it has been working fine — so no problems with the
actual code have developed over these first 3+ months of use.
This is a good thing to know.

I’ll let Brett take over now.

[ncoghlan]

Added Ned to the nosy list.

For Python core, we should also keep in mind that we do have the option of adding a -X coverage option to avoid the need for the encodings module hack.

[nedbat]

The tip of the coverage.py repo (https://bitbucket.org/ned/coveragepy) has a new implementation of the encodings hack which seems to work well.

Of course, an option to run a module before anything else in the interpreter would make the whole thing much simpler.

[brettcannon]

Brandon, can I go ahead and close this and consider the script you and Ned have in the coverage.py repo to be the canonical script to use?

[ncoghlan]

Do we really want to enshrine shadowing the encodings module as the one true way to do this kind of thing?

Although I guess defining a way to do it properly would be a fairly major undertaking, so perhaps blessing the encodings hack is the practical way forward...

If we do that, I'd like to see a devguide patch that better explains how to generate your own coverage data for the stdlib with coverage.py, the encodings hack and the standard library coverage option.

[brandon-rhodes]

Brett, yes, you are welcome to close this issue — Ned quite handily convinced me that coverage code belongs in the "coverage" distribution, not languishing about in the CPython source tree. That solution also quite beautifully solves the copyright problem. So I happily withdraw my request for this feature.

Nick, Brett is working on exactly the sort of devguide improvement that you suggest — not least because the devguide will now need to instruct people in how to build "coverage" so that its C-accelerated tracer is available, which Ned's own patch to "coverage" to cover stdlib tracing uses instead of the Python tracer that I cut-and-pasted into this patch.

Finally, it would be wonderful to have a more formal mechanism for boot-time interventions. I have mentioned before my wish that Python's first action be to open() the executable, check its tail to see if it's a valid zipfile, and if so to try loading and running "startup.py" from that zipfile. Among other things, that would allow single-file distribution of pure-Python applications without the py2exe/py2app mess that prevails in the projects I work with today. But since the whole issue of grabbing control at boot time raises hackles ("why would you want to do that!?"), and I needed something working immediately during the PyCon sprint, I elected to simply adopt "encodings.py" as my way in. It works great, and "coverage" can evolve to an even better mechanism as soon as one becomes available, should anyone want to take the bootup option and run with it.

One final thought: should PyPy etc also implement the same boot protocol, should one be invented, so that all mainline interpreters can be instrumented the same way?

[Yhg1s]

For what it's worth, the need for a bootstrap-module has also come up within Google, where we have... somewhat different requirements than most. In order to fix import paths in a way that works even when using "python -S", I had a need to patch the very first module that gets imported. Unfortunately the actual first module that gets imported depends on the arguments passed to Python. So, I wrote the attached stdlib_landmark.diff patch, which makes a 'stdlib_landmark.py' file that is both the stdlib landmark (used by Python to find the stdlib itself, currently 'os.py') *and* the very first module that is imported, always.

(The 'stdlib_landmark' name may be a little clunky; a better name, mirroring sitecustomize, may be "stdlibcustomize" or such.)

And yes, this could have been fixed in other ways. I could've patched the interpreter directly, or made Python understand symlinks to .py/.pyc files better, or tried to make a bunch of other tools work better with symlinks. This has turned out the most convenient solution for a number of reasons, though.

[brettcannon]

Going to close this and open a separate feature request to better control what module is imported first.