Enable coverage measurement of code run by multiprocessing.Process

[nedbat]

Originally reported by Ram Rachum (Bitbucket: coolrr, GitHub: coolrr)

---

If you have a multiprocessing.Process that runs on some lines of code, Coverage currently doesn't detect that and doesn't mark these lines as executed.

A sample module that uses multiprocessing.Process is attached; when run with coverage, all the 1 + 1 lines are marked as not executed, when in fact they have been executed by the auxiliary process.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/117
• This issue had attachments: coverage_multiprocessing.py. See the original issue for details.

[nedbat]

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)

---

Issue #167 was marked as a duplicate of this issue.

[nedbat]

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)

---

I am also experiencing this problem. I also tried adding the following lines to the run() method in the minimal working example posted by Ram:

import coverage
coverage.process_startup()

Unfortunately, this did not help.

[nedbat]

Original comment by amcnabb8 (Bitbucket: amcnabb8, GitHub: Unknown)

---

Hmm. I don't have a very complete picture for how coveragepy works, but it looks like when multiple processes share a coverage file, each process clobbers the file when it writes. I had expected to see something like file locking to prevent race conditions, but it looks like this doesn't happen.

One possible solution would be to have a .coverage directory (instead of a .coverage file), and to have each process write to its own file within the directory. The directory could be cleared as the main process starts, and reporting could combine the data from all of the files in the directory.

[nedbat]

Original comment by memedough (Bitbucket: memedough, GitHub: memedough)

---

Hi,

Multiprocessing uses fork, the child process has same coverage obj as parent so clobber.

pytest-cov, nose-cov, nose2-cov use multiprocessing hook after fork to child start coverage with unique data file, also use multiprocessing finalizer to stop then write file. Use them to get multiprocessing coverage out of box.

If coverage start doing multiprocessing then please let know so I make plugins works with changed coverage. Look at cov-core near top the file if you like.

:)

[nedbat]

Original comment by Anton Löfgren (Bitbucket: cath, GitHub: cath)

---

I'm experiencing some weirdness when running coverage.py on a test suite which uses multiprocessing.Process. Namely, it gets confused with regards to the tracer functions resulting in:

Coverage.py warning: Trace function changed, measurement is likely wrong: <bound method PyTracer._trace of <coverage.collector.PyTracer object at 0x2b98b0626390>>

Removing the call to threading.settrace() (in coverage.collector.Collector) seems to resolve the issue, though I'm not sure about the reliability of the coverage measurements reported.

[nedbat]

Anton: do you have a reproducible test case you could share?

[nedbat]

Original comment by Jan-Philip Gehrcke (Bitbucket: jgehrcke, GitHub: jgehrcke)

---

You should look into http://pypi.python.org/pypi/nose-cov/ which supports tracking of code run in child processes spawned via multiprocessing or subprocess modules.

[nedbat]

Original comment by eduardo schettino (Bitbucket: schettino72, GitHub: schettino72)

---

I solved this by monkey-patching the multiprocessing module.
see http://blog.schettino72.net/posts/python-code-coverage-multiprocessing.html

I wondering if this should be included in coverage, probably not by default but
activated by a flag from the command line.
$ coverage run --multiprocessing

[nedbat]

Issue #338 was marked as a duplicate of this issue.

[nedbat]

@schettino72 I couldn't get your monkey-patching code to work. Do you have a complete example?

[nedbat]

Original comment by eduardo schettino (Bitbucket: schettino72, GitHub: schettino72)

---

Hi Ned. Thanks for looking into this.

The patch I actually use is https://github.com/pydoit/doit/blob/master/tests/conftest.py#L146
It is a bit different from the blog post... I must have updated the code but didnt update the blog post.

I just tested with both pyhon 2.7 and 3.4 using coverage 3.7.1,
and it works fine for my project.
I run coverage in that project using doit coverage.

[nedbat]

@schettino72 Thanks, that works for me also, if I don't forget to use "coverage combine"! :)

[nedbat]

Original comment by Anonymous

---

I found a hole in the patch while I was working with it. It worked quite well, but processes started under coverage typically didn't take a coveragerc file with them, especially a custom one without that name.

This is the version I used;

#!python

def coverage_multiprocessing_process(): # pragma: no cover
    try:
        import coverage as _coverage
        _coverage
    except:
        return

    from coverage.collector import Collector
    from coverage import coverage
    # detect if coverage was running in forked process
    if Collector._collectors:
        original = multiprocessing.Process._bootstrap
        class Process_WithCoverage(multiprocessing.Process):
            def _bootstrap(self):
                cov = coverage(data_suffix=True,config_file=os.environ['COVERAGE_PROCESS_START'])
                cov.start()
                try:
                    return original(self)
                finally:
                    cov.stop()
                    cov.save()
        return Process_WithCoverage

ProcessCoverage = coverage_multiprocessing_process()
if ProcessCoverage:
    multiprocessing.Process = ProcessCoverage

Just a tiny improvement I thought I'd share with the community. Hope it helps.

[nedbat]

@mezarim thanks, can you explain it a little more? You are using an unusually-named configuration file?

[nedbat]

This monkeypatching is implemented in coverage.py as of f84263e515c2 (bb). I'll wait to close the issue until I can implement the config_file improvement also.

[nedbat]

Original comment by Pieter Rautenbach (Bitbucket: parautenbach, GitHub: parautenbach)

---

I've experimented with this branch, and it works great. It would be great to see this incorporated into a stable (non-alpha) release, with a .coveragerc configuration option.

[nedbat]

Original comment by Pieter Rautenbach (Bitbucket: parautenbach, GitHub: parautenbach)

---

While experimenting with this patch, I was wondering whether the multiprocessing.dummy wrapper could be used as an alternative. I'm suggesting this without having investigated its feasibility, but it could be possible to replace e.g. import multiprocessing with import multiprocessing.dummy as multiprocessing. The disadvantage is that the code under testing must be modified, but the advantage is that the standard coverage.py tool would do the job. Just a thought...

[nedbat]

Original comment by hugovk (Bitbucket: hugovk, GitHub: hugovk)

---

I'd like to start using nose's parallel testing to speed up Pillow's tests on Travis CI.

It looks like the --include=diris ignored with --concurrency=multiprocessing.

Currently it does something like this (using Coverage 3) (before some later processing to include C coverage, but let's ignore that for now).

coverage erase
coverage run --append --include=PIL/* selftest.py
coverage run --append --include=PIL/* -m nose -vx Tests/test_*.py

coverage report only lists files under PIL/ as expected, coverage is 79% (1903 missed of 9119). For example:
https://travis-ci.org/hugovk/Pillow/jobs/61138890#L2408

To use nose's parallel testing, I added --processes=-1 --process-timeout=600 to nose, and then (using Coverage 4.0a5) add --concurrency=multiprocessing to coverage, and run coverage combine at the end:

coverage erase
coverage run --append --include=PIL/* selftest.py
coverage run --append --concurrency=multiprocessing --include=PIL/* -m nose -vx Tests/test_*.py --processes=-1 --process-timeout=600
coverage combine

Nose runs the tests in parallel, they take under 100 seconds instead of over 300 seconds.

I'm sure those appends aren't quite right, but anyway, it seems --include=PIL/* is being ignored as coverage report gives 43% coverage, 22502 of 39533 lines missed, and also lists files under Tests/ and /home/travis/virtualenv/python2.7.9/ which isn't desired. For example:
https://travis-ci.org/hugovk/Pillow/jobs/61133954#L2621

[nedbat]

Issue #446 was marked as a duplicate of this issue.

[nedbat]

I'm going to mark this issue as fixed, now that 4.0 has --concurrency=multiprocessing. @hugovk If your situation is still a problem, could you open a new issue?

[nedbat]

Original comment by Andrey Cizov (Bitbucket: andreycizov, GitHub: andreycizov)

---

--concurrency=multiprocessing didn't seem to work for Pools in 4.3 (the coverage never started)

Here's the gist that fixes it:

https://gist.github.com/andreycizov/ee59806a3ac6955c127e511c5e84d2b6

[nedbat]

@andreycizov Would you mind opening a new issue, with details of the problem? A reproducible test case would be great, and also information about whether this was always a problem, or is it a regression? Thanks.