Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support coverage for py_test #43

Closed
hwright opened this issue Nov 20, 2017 · 19 comments
Closed

Support coverage for py_test #43

hwright opened this issue Nov 20, 2017 · 19 comments
Assignees
@thundergolfer
Labels
P4 type: feature request

Comments

@hwright
Copy link
Contributor

hwright commented Nov 20, 2017

bazel coverage should generate a coverage report for the python tests it runs. Ideally, it gathers the list of dependencies, and then computes coverage over that for a given test. When multiple tests are run in the same Bazel invocation, their reports should be merged into a single one.

I'm dropping this here mostly as a python-specific version of bazelbuild/bazel#1118. I know that the py_test rule is currently implemented natively in Bazel, and the implementation in rules_python just forwards there, but this is fundamentally a python issue, so this repo feels appropriate.

The general plan is to modify the python test runner within Bazel to use coverage.py to generate reports when running under coverage. I suspect it is a journey fraught with peril.

Alternatively, if it is possible to invoke coverage.py in ways other than modifying core Bazel, I'm open to suggestions.
@mattmoor
Copy link
Contributor

@damienmg assigning to you to redirect appropriately.

@hwright
Copy link
Contributor Author

hwright commented Nov 27, 2017

fwiw, I've also been poking at this as documented here: https://groups.google.com/d/msg/bazel-discuss/LC97zBrMG84/cfVNdus-AwAJ

It's been slow going, as I'm pretty unfamiliar with Bazel internals, and haven't been able to get the ear of somebody who is.

@damienmg damienmg removed their assignment Dec 14, 2017
@hwright hwright mentioned this issue Jan 8, 2018
Closed
@brandjon brandjon added P4 and removed untriaged labels Nov 9, 2018
@brandjon
Copy link
Member

brandjon commented Nov 9, 2018

Setting P4 because this isn't a blocker for 1.0 support.

@JoshSullivan11
Copy link

Has this feature been implemented yet? If so, in what release version? It seems like coverage has been able to cover every other bit of our code repo except for all python files which is makes up most of our code base. Please let me know!

@siddharthab
Copy link
Contributor

Is there a way to make bazel coverage //some_py_test not fail even if it does not collect coverage information. Currently, it fails because :lcov_merger attribute is not defined in py_test rules.

This is kind of important because we want to be able to run bazel coverage ... on a monorepo.

@gabrieldemarmiesse gabrieldemarmiesse mentioned this issue Feb 5, 2020
Closed
@dmadisetti dmadisetti mentioned this issue May 18, 2020
Open
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Aug 19, 2020
@gergelyfabian
Add pip_data_exclude to pip_repository (bazelbuild#43)
3e1a6a5
@sgammon
Copy link

sgammon commented Sep 6, 2020

how is it possible that coverage, python, and bazel don't work together, in the year of our lord 2020

@thundergolfer
Copy link
Collaborator

IIUC this problem lies in bazelbuild/bazel, and we need to work to get bazelbuild/bazel#10660 triaged there.

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

@github-actions github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Apr 28, 2021
@EnergySRE
Copy link

We at Camus Energy still want this. Python doesn't scale well without good test coverage, and coverage reports are valuable leverage.

If the author of an eventual PR is ever in San Francisco, we will buy you a taco, and tell you all about how weird it is to develop software in the energy sector. (Hopefully these words of support count as activity.)

@github-actions github-actions bot removed the Can Close? Will close in 30 days if there is no new activity label Apr 29, 2021
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

@github-actions github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Oct 27, 2021
@EnergySRE
Copy link

Boop. We still desire this 🙂

@mihaigalos
Copy link

Boop. We still desire this 🙂

Then implement it, perhaps? 🙃

@github-actions github-actions bot removed the Can Close? Will close in 30 days if there is no new activity label Oct 28, 2021
@thundergolfer thundergolfer self-assigned this Nov 1, 2021
@nstng nstng mentioned this issue Jan 14, 2022
Closed
@TLATER TLATER mentioned this issue Jan 31, 2022
Closed
5 tasks
@adam-azarchs
Copy link

At this point upstream coveragepy has lcov support. I think the main piece that's missing is that there needs to be a way to specify a target for coveragepy in py_runtime for hermetic python toolchains.

@caseyduquettesc caseyduquettesc mentioned this issue Aug 3, 2022
Open
@aignas aignas mentioned this issue Jan 9, 2023
Merged
12 tasks
aignas added a commit to aignas/rules_python that referenced this issue Jan 9, 2023
@aignas
feat(coverage): Register coverage.py to hermetic toolchains
6b54226 
Summary:
- A script to update URLs and sha256 values used to download platform
  specific coverage.py wheels.
- Allow to specify `coverage_tool` attribute in `python_repository`.
- Register `coverage.py` when running `pip_install_dependencies`
  function.
- Ensure that `use_repo` includes coverage tool, so that `bzlmod` users
  can use it too.
- Have a boolean flag, which disables setting `coverage_tool` in the
  toolchain code.

Does not work/bad/ugly:
- Use coverage.py v6.5.0 because the latest has `types.py` in the
  package directory, which imports from Python's stdlib `types` [1].
  Somehow the Python interpreter is thinking that the
  `from types import FrameType` is referring to the currently
  interpreted file and everything breaks. I would have expected the
  package to use absolute imports and only attempt to import from
  `coverage.types` if we use `coverage.types` and not just a plain
  `types` import.
- The `multi_python_versions` example cannot show coverage for the more
  complex tests that are using `subprocess`. I am wondering if this is
  related to the fact that we are including `coverage.py` via the
  toolchain and not through other mechanisms [2].
- The `__init__.py` files in the root of the WORKSPACE in `bzlmod` is
  breaking, when running under `bazel coverage //:test`. However, it
  started working when I renamed `__init__.py` to `lib.py`. I am
  suspecting that this has to do with the fact that the layer of
  indirection that `coverage` introduces could be something to do with
  that.

Work towards bazelbuild#43.

[1]: https://github.com/nedbat/coveragepy/blob/master/coverage/types.py
[2]: https://bazel.build/configure/coverage
@chrislovecnm
Copy link
Collaborator

@aignas do we have more work to do on this?

@sgammon
Copy link

sgammon commented Aug 20, 2023

this issue has been open for 6 years

@aignas
Copy link
Collaborator

aignas commented Aug 20, 2023

I think we can safely close it now as we have docs and there are quite a few versions supporting toolchain with coverage.py enabled.

@sgammon
Copy link

sgammon commented Aug 20, 2023

@aignas I just set up a brand new project on top of rules_python, using Bazel 7 and Bzlmod, and coverage still doesn't work out of the box.

Where do you see toolchains supporting coverage?

@adam-azarchs
Copy link

https://github.com/bazelbuild/rules_python/blob/main/docs/coverage.md

@aignas aignas closed this as completed Nov 5, 2023
@aignas
Copy link
Collaborator

aignas commented Nov 5, 2023

See https://rules-python.readthedocs.io/en/latest/coverage.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thundergolfer thundergolfer

Labels
P4 type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests