Coverage measurement fails on code containing os.exec* methods

[nedbat]

Originally reported by Anonymous

---

I recently tried to measure coverage of a program that calls os.execvpe, essentially causing the process to be replaced by a different one. This did not record any coverage information at all.

The reason of course is that os.execvpe does not return, so there is no opportunity to call coverage.stop() and coverage.save() as is done if e.g. an exception is thrown. I'd suggest this method could be "monkey-patched" so that such code can be inserted before it.
(and also the other 7 os.exec* methods of course)

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/43
• This issue had attachments: execfunctions.bundle. See the original issue for details.

[nedbat]

Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)

---

Sorry, didn't mean to report this anonymously.

[nedbat]

Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)

---

I have a patch for this one also if you're interested.

[nedbat]

I'd be interested to see a patch. It sounds very involved!

[nedbat]

Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)

---

Patch attached against current trunk. Could be simplified if it assumed that these functions called each other internally, which they do currently.

Tested with a small program as follows

#!python
import os

print "First program..."
os.execvp("./prog2.py", [ "./prog2.py", "-x" ])

which I can now get coverage information out of.

[nedbat]

Original comment by NiklasH (Bitbucket: nh2, GitHub: nh2)

---

What happened to this?

[nedbat]

I haven't applied the patch yet, because I'd only heard one request for it (this ticket), and I am averse to monkeypatching. But I now have a second request, and this is a fairly small patch.

Interestingly, if execvpe would execute the atexit-registered handlers before changing the process over, it would just work. I created http://bugs.python.org/issue16822 to request Python to be fixed.

[nedbat]

Considering this for the next release.

[nedbat]

For anyone looking for Geoff's changes:

diff -r [f7d26908601c (bb)](https://bitbucket.org/ned/coveragepy/commits/f7d26908601c) -r [f3a76cf7aa00 (bb)](https://bitbucket.org/ned/coveragepy/commits/f3a76cf7aa00) coverage/control.py
--- a/coverage/control.py       Sun Nov 07 19:45:54 2010 -0500
+++ b/coverage/control.py       Mon Nov 15 21:36:31 2010 +0100
@@ -360,6 +360,14 @@

         self._harvested = False
         self.collector.start()
+        os.execvpe = self.intercept(os.execvpe)
+
+    def intercept(self, method):
+        def new_method(*args, **kw):
+            self.stop()
+            self.save()
+            method(*args, **kw)
+        return new_method

     def stop(self):
         """Stop measuring code coverage."""

and then:

diff -r [f3a76cf7aa00 (bb)](https://bitbucket.org/ned/coveragepy/commits/f3a76cf7aa00) -r [ba05ad03668e (bb)](https://bitbucket.org/ned/coveragepy/commits/ba05ad03668e) coverage/control.py
--- a/coverage/control.py       Mon Nov 15 21:36:31 2010 +0100
+++ b/coverage/control.py       Mon Nov 15 22:37:22 2010 +0100
@@ -359,15 +359,13 @@
             self.omit_match = FnmatchMatcher(self.omit)

         self._harvested = False
+        for funcName in [ 'execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe' ]:
+            newFunc = self.intercept(getattr(os, funcName))
+            setattr(os, funcName, newFunc)
         self.collector.start()
-        os.execvpe = self.intercept(os.execvpe)

-    def intercept(self, method):
-        def new_method(*args, **kw):
-            self.stop()
-            self.save()
-            method(*args, **kw)
-        return new_method
+    def intercept(self, method):
+        return StopCoverageDecorator(self, method)

     def stop(self):
         """Stop measuring code coverage."""
@@ -612,6 +610,21 @@
         return info


+class StopCoverageDecorator:
+    inDecorator = False
+    def __init__(self, cov, method):
+        self.cov = cov
+        self.method = method
+
+    def __call__(self, *args, **kw):
+        if not StopCoverageDecorator.inDecorator:
+            StopCoverageDecorator.inDecorator = True
+            self.cov.stop()
+            self.cov.save()
+        self.method(*args, **kw)
+        StopCoverageDecorator.inDecorator = False
+
+
 def process_startup():
     """Call this at Python startup to perhaps measure coverage.

[nedbat]

Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)

---

Looking at this again. There are other circumstances where no coverage information is produced because the atexit handlers are not called:

1. When processes are terminated by a signal
2. When control is not returned to the Python interpreter. I've run into this in several circumstances under Jython, where Java code shuts down the JVM without returning control to Jython or calling any exit handlers.

In the light of this, I'm wondering if a generic, if slightly clumsy, solution would be simply to provide a "process_shutdown" method (in a similar way to process_startup) so that this external handler could be called explicitly in any of these circumstances.

Then it's easy to add code like

#!python

try:
    import coverage
    coverage.process_shutdown()
except:
    pass

and call it in signal handlers, just before os.exec* or in a JVM exit handler to work around all these cases.

Would be better of course if it was detected automatically, but this will avoid the evils of monkey patching and is more generic than the code I posted here. Can provide a patch if you agree to the solution.

[nedbat]

Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)

---

Implementing a solution like this now appears to be difficult, as the "coverage" object is not necessarily stored in a global variable any more.

I resorted to the following in my code, which does not require patching coverage at all, but only works under Python 2 and relies on an undocumented feature of Python's atexit module:

#!python

import atexit
for func, args, kw in atexit._exithandlers:
    if func.__module__.startswith("coverage."):
        func(*args, **kw)

Clearly not ideal but about the only thing I could think of to get my coverage measurement working with the latest version.

[spaceone]

CC.
Still affected in ```
Python 3.7.3 (default, Jul 25 2020, 13:03:44)

import coverage
coverage.version
'4.5.2'

[rgbyrnes]

Here's another hack that seems to work for Python 3, at least with coverage run: it walks the stack to find the Coverage object ...

#!python

if 'COVERAGE_RUN' in os.environ and sys.gettrace() is not None:
    def _find_cov():
        from traceback import walk_stack
        for (f, _) in walk_stack(None):
            for obj in f.f_locals.values():
                if (obj.__class__.__name__ == 'CoverageScript' and
                    obj.__module__ == 'coverage.cmdline'):
                    return obj.coverage
    cov = _find_cov()
    if cov is not None:
        cov.stop()
        cov.save()

[sigma67]

What's the 2023 solution to this issue with latest coverage.py?

[nedbat]

Nothing new has happened. Can you tell us more about how and why os.exec is involved? It's a difficult thing to put a leash around with certainty.

[sigma67]

I have a function that calls os.execv, which I'm testing with pytest, let's call it func_execv.

My testing code looks as follows (to prevent pytest crashing when the process is replaced). Minimal reproduction:

from multiprocessing import Process
import os


def func_execv(cmd, args):
    os.execvp(cmd, args)


def test_func_execv(capfd):
    cmd = "echo"
    args = [cmd, "test"]
    p = Process(target=func_execv, args=(cmd, args))
    p.start()
    p.join()
    assert "test" in capfd.readouterr().out

Run this file with pytest file.py --cov in an environment with pytest and pytest-cov.

I have to execute the func_execv inside Process because otherwise pytest crashes due to process exit.

The code in func_execv is not covered. coverage report -m:

Name            Stmts   Miss  Cover   Missing
---------------------------------------------
scratch_51.py      11      1    91%   6
---------------------------------------------
TOTAL              11      1    91%

[nedbat]

Do the steps for subprocess measurement help? https://coverage.readthedocs.io/en/7.3.2/subprocess.html

[sigma67]

I've added the suggested line to _virtualenv.pth and also set the env variable COVERAGE_PROCESS_START, but it didn't help.

Can you suggest a basic setup with the minimum reproduction example where it would work?