Enable coverage for subprocesses #6546

adswa · 2022-03-14T12:52:31Z

The changes here enable subprocess coverage and fix #6367. They follow different docs:

Following https://coverage.readthedocs.io/en/latest/subprocess.html, this change sets the COVERAGE_PROCESS_START environment variable, and lets it point to the .coveragerc file.
Following the first approach described in https://coverage.readthedocs.io/en/latest/subprocess.html, this change creates a sitecustomize.py file. As this file needs to be in the Pythonpath, I'm following https://pymotw.com/2/site/#sitecustomize to modify the PYTHONPATH accordingly
Finally, this change changes the installation of DataLad from a system-wide installation to a local (user) installation. This is a workaround, because with a system-wide installation coverage refused to collect any coverage for datalad or other packages in site-packages as it reported them to be "in the stdlib" (see details below). We should discuss if this is a problem - was there any reason that the installation is done system-wide? Is there any problem doing it only per user (pip install . --user)?

........S...................sys.path:
    /home/travis/build/datalad/datalad/__testhome__
    /home/travis/build/datalad/datalad/__testhome__/tools/coverage-bin/sitecustomize.py
    /tmp/dl-miniconda-lwdfcfx3/lib/python39.zip
    /tmp/dl-miniconda-lwdfcfx3/lib/python3.9
    /tmp/dl-miniconda-lwdfcfx3/lib/python3.9/lib-dynload
    /tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages
Python stdlib matching: <TreeMatcher pylib ['/tmp/dl-miniconda-lwdfcfx3/lib/python3.9']>
Coverage code matching: <TreeMatcher coverage ['/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/coverage']>
Third-party lib matching: <TreeMatcher third ['/home/travis/.local/bin', '/home/travis/.local/lib/python3.9/site-packages', '/tmp/dl-miniconda-lwdfcfx3/bin', '/tmp/dl-miniconda-lwdfcfx3/lib/python', '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages']>
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/threading.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/coverage/execfile.py': is part of coverage.py
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/genericpath.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/coverage/files.py': is part of coverage.py
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/posixpath.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/coverage/python.py': is part of coverage.py
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/coverage/phystokens.py': is part of coverage.py
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/tokenize.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/bin/datalad': is a third-party module
Not tracing '<frozen importlib._bootstrap>': not a real original file name
Not tracing '<frozen importlib._bootstrap_external>': not a real original file name
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/_collections_abc.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/os.py': is in the stdlib
Not tracing '<frozen zipimport>': not a real original file name
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/config.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/logging/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/string.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/re.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/enum.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/types.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/sre_compile.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/sre_parse.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/_weakrefset.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/weakref.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/fasteners/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/__future__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/fasteners/lock.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/fasteners/_utils.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/contextlib.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/functools.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/fasteners/process_lock.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/fasteners/version.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/pathlib.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/urllib/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/urllib/parse.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/collections/__init__.py': is in the stdlib
Not tracing '<string>': not a real original file name
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/abc.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/consts.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/coreprotocols.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/protocol.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/base_events.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/concurrent/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/concurrent/futures/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/concurrent/futures/_base.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/ssl.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/constants.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/coroutines.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/base_futures.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/format_helpers.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/log.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/events.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/contextvars.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/exceptions.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/base_tasks.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/futures.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/protocols.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/sslproto.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/transports.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/staggered.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/locks.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/tasks.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/typing.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/trsock.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/runners.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/queues.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/streams.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/subprocess.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/threads.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/unix_events.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/base_subprocess.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/asyncio/selector_events.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/utils.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/tempfile.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/gzip.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/support/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/support/exceptions.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/exception.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/platform.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/gitrunner.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/dochelpers.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/runner.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/queue.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/runner/runnerthreads.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/shutil.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/subprocess.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/warnings.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/log.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/logging/handlers.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/support/ansi_colors.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/ui/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/ui/dialog.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/getpass.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/ui/base.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/interface/__init__.py': is in the stdlib
Not tracing '/tmp/dl-miniconda-lwdfcfx3/lib/python3.9/site-packages/datalad/interface/common_cfg.py': is in the stdlib
[...]

I have tested whether this works and what an impact it would make by running only datalad.distributed tests, which include the ora_remote.py that was covered to only 33.9%:

With the changes in this PR, coverage of ora_remote.py seems to be bumped by +46% to 80% coverage:

The changes follow the documentation in coverage.readthedocs.io/subprocess.html and pymotw.com/2/site/#sitecustomize. They are based on a config variable switch to enable subprocess coverage reporting, and the introduction of a sitecustomize.py script, that performs a coverage process startup for non-datalad processes. I encountered unexpected behavior in that a system-wide installation of DataLad interfered with coverage reports - I suspect this is because I altered the PYTHONPATH variable to make the sitecustomize.py script discoverable. This unexpected behavior manifested in coverage debug messages stating that DataLad and other packages installed in site-packages would be part of stdlib, which coverage by default would not trace. I worked around this by changing datalad's installation to a user-based installation. Maybe this could also be resolved by placing sitecustomize.py into a path that already is in the Pythonpath (Lib/), but I did not go down this route. Fixes datalad#6367

adswa · 2022-03-15T10:35:44Z

ahh, I just noticed that this only enables this for Travis. Maybe we can also do it for appveyor.

…bprocesses

adswa · 2022-03-15T11:28:35Z

The changes on travis pass over all checks, and bump coverage especially in ora_remote.py. Overall project coverage increases to 91%. I'm cancelling tests again with the exception of Appveyor, as I now try to make appveyor collect subprocess coverage, too.

.travis.yml

codeclimate · 2022-03-15T12:48:17Z

Code Climate has analyzed commit bc34a9d and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Security	1

View more on Code Climate.

adswa · 2022-03-15T12:50:12Z

Things seems to work for appveyor, too.

adswa · 2022-03-16T08:30:09Z

The test failures are all known, except for one, that is a stalling CI run that looks unrelated to this change.

yarikoptic · 2022-03-16T11:43:09Z

Didn't see stalling recently so wouldn't be sure that it is unrelated (didn't look in detail). Does it work if restarted?

yarikoptic · 2022-03-16T11:44:40Z

FWIW restarted Travis

mih

This looks good to me and well worth when exploring DataLad's blind spots.

It did stall/slow on Ubuntu (appveyor) too, though (but only one run) -- so I restarted that one.

Otherwise, good to go, I think. Thx!

adswa · 2022-03-17T10:13:33Z

The appveyor failures are the known stuff.

adswa force-pushed the wip-cov branch 15 times, most recently from d663979 to 43ed0e8 Compare March 15, 2022 09:21

adswa force-pushed the wip-cov branch from 43ed0e8 to db25620 Compare March 15, 2022 10:14

adswa changed the title ~~WIP: try to get subprocess coverage~~ Enable coverage for subprocesses Mar 15, 2022

adswa added the semver-tests Changes only affect tests, no impact on version label Mar 15, 2022

TST: Reuse test coverage helpers on non-Win Appveyor runs to cover su…

6c6ce1f

…bprocesses

yarikoptic reviewed Mar 15, 2022

View reviewed changes

.travis.yml Show resolved Hide resolved

adswa force-pushed the wip-cov branch from 6577591 to bc34a9d Compare March 15, 2022 12:46

adswa force-pushed the wip-cov branch from bc34a9d to 6c6ce1f Compare March 15, 2022 12:48

mih approved these changes Mar 17, 2022

View reviewed changes

adswa merged commit ce105d1 into datalad:master Mar 17, 2022

adswa deleted the wip-cov branch March 17, 2022 10:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable coverage for subprocesses #6546

Enable coverage for subprocesses #6546

adswa commented Mar 14, 2022 •

edited

Loading

adswa commented Mar 15, 2022

adswa commented Mar 15, 2022

codeclimate bot commented Mar 15, 2022

adswa commented Mar 15, 2022

adswa commented Mar 16, 2022

yarikoptic commented Mar 16, 2022

yarikoptic commented Mar 16, 2022

mih left a comment •

edited

Loading

adswa commented Mar 17, 2022

Enable coverage for subprocesses #6546

Enable coverage for subprocesses #6546

Conversation

adswa commented Mar 14, 2022 • edited Loading

adswa commented Mar 15, 2022

adswa commented Mar 15, 2022

codeclimate bot commented Mar 15, 2022

adswa commented Mar 15, 2022

adswa commented Mar 16, 2022

yarikoptic commented Mar 16, 2022

yarikoptic commented Mar 16, 2022

mih left a comment • edited Loading

Choose a reason for hiding this comment

adswa commented Mar 17, 2022

adswa commented Mar 14, 2022 •

edited

Loading

mih left a comment •

edited

Loading