Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable coverage measurement of code run by multiprocessing.Process #117

Closed
nedbat opened this issue Feb 2, 2011 · 22 comments
Closed

Enable coverage measurement of code run by multiprocessing.Process #117

nedbat opened this issue Feb 2, 2011 · 22 comments
Labels
enhancement New feature or request run

Comments

@nedbat
Copy link
Owner

nedbat commented Feb 2, 2011

Originally reported by Ram Rachum (Bitbucket: coolrr, GitHub: coolrr)


If you have a multiprocessing.Process that runs on some lines of code, Coverage currently doesn't detect that and doesn't mark these lines as executed.

A sample module that uses multiprocessing.Process is attached; when run with coverage, all the 1 + 1 lines are marked as not executed, when in fact they have been executed by the auxiliary process.


@nedbat
Copy link
Owner Author

nedbat commented Mar 8, 2012

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)


Issue #167 was marked as a duplicate of this issue.

@nedbat
Copy link
Owner Author

nedbat commented Mar 8, 2012

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)


I am also experiencing this problem. I also tried adding the following lines to the run() method in the minimal working example posted by Ram:

import coverage
coverage.process_startup()

Unfortunately, this did not help.

@nedbat
Copy link
Owner Author

nedbat commented Mar 8, 2012

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)


Hmm. I don't have a very complete picture for how coveragepy works, but it looks like when multiple processes share a coverage file, each process clobbers the file when it writes. I had expected to see something like file locking to prevent race conditions, but it looks like this doesn't happen.

One possible solution would be to have a .coverage directory (instead of a .coverage file), and to have each process write to its own file within the directory. The directory could be cleared as the main process starts, and reporting could combine the data from all of the files in the directory.

@nedbat
Copy link
Owner Author

nedbat commented Mar 12, 2012

Original comment by memedough (Bitbucket: memedough, GitHub: memedough)


Hi,

Multiprocessing uses fork, the child process has same coverage obj as parent so clobber.

pytest-cov, nose-cov, nose2-cov use multiprocessing hook after fork to child start coverage with unique data file, also use multiprocessing finalizer to stop then write file. Use them to get multiprocessing coverage out of box.

If coverage start doing multiprocessing then please let know so I make plugins works with changed coverage. Look at cov-core near top the file if you like.

:)

@nedbat
Copy link
Owner Author

nedbat commented Aug 20, 2012

Original comment by Anton Löfgren (Bitbucket: cath, GitHub: cath)


I'm experiencing some weirdness when running coverage.py on a test suite which uses multiprocessing.Process. Namely, it gets confused with regards to the tracer functions resulting in:

Coverage.py warning: Trace function changed, measurement is likely wrong: <bound method PyTracer._trace of <coverage.collector.PyTracer object at 0x2b98b0626390>>

Removing the call to threading.settrace() (in coverage.collector.Collector) seems to resolve the issue, though I'm not sure about the reliability of the coverage measurements reported.

@nedbat
Copy link
Owner Author

nedbat commented Sep 17, 2012

Anton: do you have a reproducible test case you could share?

@nedbat
Copy link
Owner Author

nedbat commented Nov 13, 2012

Original comment by Jan-Philip Gehrcke (Bitbucket: jgehrcke, GitHub: jgehrcke)


You should look into http://pypi.python.org/pypi/nose-cov/ which supports tracking of code run in child processes spawned via multiprocessing or subprocess modules.

@nedbat
Copy link
Owner Author

nedbat commented Dec 31, 2012

Original comment by eduardo schettino (Bitbucket: schettino72, GitHub: schettino72)


I solved this by monkey-patching the multiprocessing module.
see http://blog.schettino72.net/posts/python-code-coverage-multiprocessing.html

I wondering if this should be included in coverage, probably not by default but
activated by a flag from the command line.
$ coverage run --multiprocessing

@nedbat
Copy link
Owner Author

nedbat commented Jan 26, 2015

Issue #338 was marked as a duplicate of this issue.

@nedbat
Copy link
Owner Author

nedbat commented Jan 28, 2015

@schettino72 I couldn't get your monkey-patching code to work. Do you have a complete example?

@nedbat
Copy link
Owner Author

nedbat commented Jan 28, 2015

Original comment by eduardo schettino (Bitbucket: schettino72, GitHub: schettino72)


Hi Ned. Thanks for looking into this.

The patch I actually use is https://github.com/pydoit/doit/blob/master/tests/conftest.py#L146
It is a bit different from the blog post... I must have updated the code but didnt update the blog post.

I just tested with both pyhon 2.7 and 3.4 using coverage 3.7.1,
and it works fine for my project.
I run coverage in that project using doit coverage.

@nedbat
Copy link
Owner Author

nedbat commented Jan 28, 2015

@schettino72 Thanks, that works for me also, if I don't forget to use "coverage combine"! :)

@nedbat
Copy link
Owner Author

nedbat commented Jan 28, 2015

Original comment by Anonymous


I found a hole in the patch while I was working with it. It worked quite well, but processes started under coverage typically didn't take a coveragerc file with them, especially a custom one without that name.

This is the version I used;

#!python

def coverage_multiprocessing_process(): # pragma: no cover
    try:
        import coverage as _coverage
        _coverage
    except:
        return

    from coverage.collector import Collector
    from coverage import coverage
    # detect if coverage was running in forked process
    if Collector._collectors:
        original = multiprocessing.Process._bootstrap
        class Process_WithCoverage(multiprocessing.Process):
            def _bootstrap(self):
                cov = coverage(data_suffix=True,config_file=os.environ['COVERAGE_PROCESS_START'])
                cov.start()
                try:
                    return original(self)
                finally:
                    cov.stop()
                    cov.save()
        return Process_WithCoverage

ProcessCoverage = coverage_multiprocessing_process()
if ProcessCoverage:
    multiprocessing.Process = ProcessCoverage

Just a tiny improvement I thought I'd share with the community. Hope it helps.

@nedbat
Copy link
Owner Author

nedbat commented Jan 30, 2015

@mezarim thanks, can you explain it a little more? You are using an unusually-named configuration file?

@nedbat
Copy link
Owner Author

nedbat commented Jan 31, 2015

This monkeypatching is implemented in coverage.py as of f84263e515c2 (bb). I'll wait to close the issue until I can implement the config_file improvement also.

@nedbat
Copy link
Owner Author

nedbat commented Apr 22, 2015

Original comment by Pieter Rautenbach (Bitbucket: parautenbach, GitHub: parautenbach)


I've experimented with this branch, and it works great. It would be great to see this incorporated into a stable (non-alpha) release, with a .coveragerc configuration option.

@nedbat
Copy link
Owner Author

nedbat commented Apr 27, 2015

Original comment by Pieter Rautenbach (Bitbucket: parautenbach, GitHub: parautenbach)


While experimenting with this patch, I was wondering whether the multiprocessing.dummy wrapper could be used as an alternative. I'm suggesting this without having investigated its feasibility, but it could be possible to replace e.g. import multiprocessing with import multiprocessing.dummy as multiprocessing. The disadvantage is that the code under testing must be modified, but the advantage is that the standard coverage.py tool would do the job. Just a thought...

@nedbat
Copy link
Owner Author

nedbat commented May 4, 2015

Original comment by hugovk (Bitbucket: hugovk, GitHub: hugovk)


I'd like to start using nose's parallel testing to speed up Pillow's tests on Travis CI.

It looks like the --include=diris ignored with --concurrency=multiprocessing.

Currently it does something like this (using Coverage 3) (before some later processing to include C coverage, but let's ignore that for now).

coverage erase
coverage run --append --include=PIL/* selftest.py
coverage run --append --include=PIL/* -m nose -vx Tests/test_*.py

coverage report only lists files under PIL/ as expected, coverage is 79% (1903 missed of 9119). For example:
https://travis-ci.org/hugovk/Pillow/jobs/61138890#L2408

To use nose's parallel testing, I added --processes=-1 --process-timeout=600 to nose, and then (using Coverage 4.0a5) add --concurrency=multiprocessing to coverage, and run coverage combine at the end:

coverage erase
coverage run --append --include=PIL/* selftest.py
coverage run --append --concurrency=multiprocessing --include=PIL/* -m nose -vx Tests/test_*.py --processes=-1 --process-timeout=600
coverage combine

Nose runs the tests in parallel, they take under 100 seconds instead of over 300 seconds.

I'm sure those appends aren't quite right, but anyway, it seems --include=PIL/* is being ignored as coverage report gives 43% coverage, 22502 of 39533 lines missed, and also lists files under Tests/ and /home/travis/virtualenv/python2.7.9/ which isn't desired. For example:
https://travis-ci.org/hugovk/Pillow/jobs/61133954#L2621

@nedbat
Copy link
Owner Author

nedbat commented Nov 14, 2015

Issue #446 was marked as a duplicate of this issue.

@nedbat
Copy link
Owner Author

nedbat commented Nov 15, 2015

I'm going to mark this issue as fixed, now that 4.0 has --concurrency=multiprocessing. @hugovk If your situation is still a problem, could you open a new issue?

@nedbat
Copy link
Owner Author

nedbat commented Dec 28, 2016

Original comment by Andrey Cizov (Bitbucket: andreycizov, GitHub: andreycizov)


--concurrency=multiprocessing didn't seem to work for Pools in 4.3 (the coverage never started)

Here's the gist that fixes it:

https://gist.github.com/andreycizov/ee59806a3ac6955c127e511c5e84d2b6

@nedbat
Copy link
Owner Author

nedbat commented Dec 28, 2016

@andreycizov Would you mind opening a new issue, with details of the problem? A reproducible test case would be great, and also information about whether this was always a problem, or is it a regression? Thanks.

agronholm added a commit to agronholm/coveragepy that referenced this issue Aug 16, 2020
This switches the curio backend to using the native task timeout mechanism.
@nedbat nedbat added enhancement New feature or request and removed proposal labels Oct 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request run
Projects
None yet
Development

No branches or pull requests

1 participant