Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test coverage collecting #128

Open
wants to merge 1 commit into
base: master
from

Conversation

@Jamim
Copy link
Contributor

Jamim commented Sep 7, 2019

Hi everyone,

I believe that we should be able to see the test coverage.

These changes:

  • make tests use pytest
  • add displaying of the test coverage
  • add sending of collected coverage data to https://codecov.io/
  • fix skipping tests for pypy3.5

Best regards!

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Sep 7, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@9fa92f3). Click here to learn what that means.
The diff coverage is 89.65%.

Impacted file tree graph

@@           Coverage Diff            @@
##             master    #128   +/-   ##
========================================
  Coverage          ?   55.7%           
========================================
  Files             ?      52           
  Lines             ?    2953           
  Branches          ?     465           
========================================
  Hits              ?    1645           
  Misses            ?    1249           
  Partials          ?      59
Impacted Files Coverage Δ
...er/src/opentelemetry/ext/jaeger/gen/agent/Agent.py 30.3% <0%> (ø)
...r/src/opentelemetry/ext/jaeger/gen/agent/ttypes.py 100% <100%> (ø)
...try-api/src/opentelemetry/context/async_context.py 95.65% <95.65%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fa92f3...294ea63. Read the comment docs.

@Jamim

This comment has been minimized.

Copy link
Contributor Author

Jamim commented Sep 7, 2019

@reyang
reyang approved these changes Sep 7, 2019
Copy link
Contributor

reyang left a comment

LGTM.

Copy link
Collaborator

toumorokoshi left a comment

Just want to call out the pytest change. If we have a consensus there I approve.

@@ -0,0 +1,8 @@
[pytest]

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Sep 9, 2019

Collaborator

I know @c24t had some reservations about using pytest. We were supposed to discuss this after the alpha release.

FWIW I really like pytest and wholeheartedly support using pytest and pytest functionality.

This comment has been minimized.

Copy link
@Jamim

Jamim Sep 10, 2019

Author Contributor

I know @c24t had some reservations about using pytest.

Hello @c24t, can you please clarify what kind of reservations you had?
Thank you!

We were supposed to discuss this after the alpha release.

Hi @toumorokoshi,
I think nothing prevents us from discussing it a bit earlier.

Also, I found a related issue: #28.

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

My biggest complaints about pytest have to do with fixtures. Fixtures with side effects, autouse fixtures, and the monkeypatch fixture in particular have the potential to turn any test suite into a horrifying mess. Using fixtures as arguments in tests makes working with tests difficult outside of pytest, and adds a lot of magic. I don't mean to say it's not possible to write nice tests with pytest, just that the library encourages some pretty bad (IMHO) behavior.

When we considered adding it before it wasn't clear that there were any features of pytest that we couldn't get just as easily from the built in unittest library. It may be that the extra complexity is justified if pytest makes writing and running tests easier. I still don't see a strong reason for using pytest in this PR, but I may be missing something here. What makes pytest --cov better than coverage run?

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

Hello @toumorokoshi and @c24t,

On the SIG meeting 2019-09-12 we discussed migration to pytest and, if I recall correctly, we found that most developers are fine with it or have no opinion. Unfortunately, we didn't write out a corresponding decision in the OT Python SIG Notes.
So my question is, can we resolve this thread now and move forward with pytest or is it necessary to have an additional discussion?

Thank you!

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 10, 2019

Collaborator

I think I've pinged the appropriate opinionated people regarding the use of pytest, so I'm going to remove my "changes required". I'll let others who are against pytest argue their point.

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Oct 10, 2019

Member

@toumorokoshi To actually dismiss your review you have to expand the "Changes requested" box at the bottom of the PR, and then click "..." -> "Dismiss" next to your review there. Re-requesting your review won't actually dismiss it (I ran into that too once, it's not the most intuitive UI).

tox.ini Outdated
@@ -38,7 +39,10 @@ commands =
mypy: mypy --namespace-packages opentelemetry-api/src/opentelemetry/
; For test code, we don't want to enforce the full mypy strictness
mypy: mypy --namespace-packages --config-file=mypy-relaxed.ini opentelemetry-api/tests/
test: python -m unittest discover
test: pytest --cov {envsitepackagesdir}/opentelemetry

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Sep 9, 2019

Collaborator

there's a few unit tests for opentelemetry-api as well. I feel like we should report coverage on those.

This comment has been minimized.

Copy link
@Jamim

Jamim Sep 9, 2019

Author Contributor

Hello @toumorokoshi,

Coverage on API is being reported for every *-test-* due to settings in pytest.ini.
Three lines those follows this one are for additional reporting.

opentelemetry-coverage

You can find the full build log for Python 3.7 here:
https://travis-ci.org/open-telemetry/opentelemetry-python/jobs/581936178

Also, please let me know if I'm missing something or there is any misunderstanding on my side.

Thanks!

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Sep 9, 2019

Member

So the combined coverage report (e.g. considering that the ext-wsgi tests will also increase coverage of the API code) is available under https://codecov.io/github/open-telemetry/opentelemetry-python/commit/de8f21ecf59322ddbd71c145c74326824a16bd30? The API package's trace module seems to be missing in that report.

This comment has been minimized.

Copy link
@Jamim

Jamim Sep 10, 2019

Author Contributor

So the combined coverage report (e.g. considering that the ext-wsgi tests will also increase coverage of the API code) is available under codecov.io/github/open-telemetry/opentelemetry-python/commit/de8f21ecf59322ddbd71c145c74326824a16bd30?

Yes, it is.

The API package's trace module seems to be missing in that report.

I've tried to investigate and fix this, but it seems like this is a bug on the Codecov side.

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

The reason you need {envsitepackagesdir} here is because we're installing to site-packages after #122 right? This makes for some pretty ugly paths in the coverage report vs. something like:

(cd opentelemetry-sdk/tests; pytest --cov ../src)

This comment has been minimized.

Copy link
@Jamim

Jamim Sep 11, 2019

Author Contributor

I've described this bug here.

This comment has been minimized.

Copy link
@c24t

c24t Sep 16, 2019

Contributor

@Jamim can you post the text for people like me that can't log into that slack?

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

The issue is already resolved in the current version of changes, but here that text is:

Hi there,
I found that some files are not visible on Codecov.io for some reason.
I tried to investigate this but I didn't find a reason why some files are shown and other files are not.

PR: #128
Build: https://travis-ci.org/open-telemetry/opentelemetry-python/builds/581936174
Codecov: https://codecov.io/gh/open-telemetry/opentelemetry-python/tree/de8f21ecf59322ddbd71c145c74326824a16bd30

On https://codecov.io/gh/open-telemetry/opentelemetry-python/tree/de8f21ecf59322ddbd71c145c74326824a16bd30/opentelemetry-api/src/opentelemetry page I'm expecting to see

context/
distributedcontext/
logs/
metrics/
resources/
trace/
loader.py
types.py
version.py

but I can see only

context/
distributedcontext/
loader.py
types.py

And corresponding reports contain stats for missing files, e.g.
https://codecov.io/codecov/v4/raw/2019-09-07/E6CBE819EACB252EDCB65A9B1EE32DAB/de8f21ecf59322ddbd71c145c74326824a16bd30/675278e2-de86-49ca-b141-f260fb05b38c.txt contains stats for API's metrics/__init__.py.

Copy link
Member

Oberon00 left a comment

Nice work so far, I think I spotted some dead code through the report too 😃 👍

tox.ini Outdated Show resolved Hide resolved
tox.ini Outdated
@@ -15,6 +15,7 @@ python =
[testenv]
deps =
mypy: mypy~=0.711
test: pytest-cov

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Sep 9, 2019

Member

Please make the coverage a different environment: I want to run unit tests without coverage (assuming that coverage significantly slows down the unit tests, otherwise I don't really care). Maybe https://tox.readthedocs.io/en/latest/config.html#complex-factor-conditions can help, or maybe we can call the environment "TESTENVNAME-cov" and set the --cov option in a setenv to minimize code duplication in tox.ini.

This comment has been minimized.

Copy link
@Jamim

Jamim Sep 9, 2019

Author Contributor

The setup is quite complex already.
I think we shouldn't increase the complexity until we really see some performance issues.

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

Hello @Oberon00,
Eventually, I've switched to the approach that you suggested in order to use the editable mode.
Could you please re-review this PR?
Thank you!

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

Why is editable mode required? Is that to more quickly iterate on unit tests?

I think it would be great to also benchmark the performance of the unit tests with coverage. In my experience it's not a huge overhead.

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 11, 2019

Author Contributor

Why is editable mode required?

Answered in #128 (comment).

Is that to more quickly iterate on unit tests?

I didn't check, but it's not a reason anyway.

tox.ini Outdated
@@ -38,7 +39,10 @@ commands =
mypy: mypy --namespace-packages opentelemetry-api/src/opentelemetry/
; For test code, we don't want to enforce the full mypy strictness
mypy: mypy --namespace-packages --config-file=mypy-relaxed.ini opentelemetry-api/tests/
test: python -m unittest discover
test: pytest --cov {envsitepackagesdir}/opentelemetry

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Sep 9, 2019

Member

So the combined coverage report (e.g. considering that the ext-wsgi tests will also increase coverage of the API code) is available under https://codecov.io/github/open-telemetry/opentelemetry-python/commit/de8f21ecf59322ddbd71c145c74326824a16bd30? The API package's trace module seems to be missing in that report.

@Jamim Jamim changed the title Add test coverage collecting [WIP] Add test coverage collecting Sep 11, 2019
Copy link
Contributor

c24t left a comment

The codecov.io integration looks great. Did you have to configure anything outside this repo to get it working? If there are project-specific settings there, who has access to them?

tox.ini Outdated
@@ -38,7 +39,10 @@ commands =
mypy: mypy --namespace-packages opentelemetry-api/src/opentelemetry/
; For test code, we don't want to enforce the full mypy strictness
mypy: mypy --namespace-packages --config-file=mypy-relaxed.ini opentelemetry-api/tests/
test: python -m unittest discover
test: pytest --cov {envsitepackagesdir}/opentelemetry

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

The reason you need {envsitepackagesdir} here is because we're installing to site-packages after #122 right? This makes for some pretty ugly paths in the coverage report vs. something like:

(cd opentelemetry-sdk/tests; pytest --cov ../src)
pytest.ini Outdated
@@ -0,0 +1,8 @@
[pytest]
addopts =
--cov-append

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

Why put these here instead of tox.ini? -rsv seems sane, but I'd be pretty surprised if I appended to the coverage report instead of regenerating it by running pytest --cov, e.g. after switching branches.

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

Fair enough. There is only -rs -v in addopts now.

tox.ini Outdated
@@ -38,7 +39,10 @@ commands =
mypy: mypy --namespace-packages opentelemetry-api/src/opentelemetry/
; For test code, we don't want to enforce the full mypy strictness
mypy: mypy --namespace-packages --config-file=mypy-relaxed.ini opentelemetry-api/tests/
test: python -m unittest discover
test: pytest --cov {envsitepackagesdir}/opentelemetry
test-sdk: coverage report --include *opentelemetry/sdk/*

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

Why filter the results after running coverage instead of passing the path to pytest? E.g.

pytest --cov-report xml --cov ../src/

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

It was necessary in purpose to collect coverage not only for a current one package, but for api/sdk as well, but now those extra reports are gone.

@@ -0,0 +1,8 @@
[pytest]

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

My biggest complaints about pytest have to do with fixtures. Fixtures with side effects, autouse fixtures, and the monkeypatch fixture in particular have the potential to turn any test suite into a horrifying mess. Using fixtures as arguments in tests makes working with tests difficult outside of pytest, and adds a lot of magic. I don't mean to say it's not possible to write nice tests with pytest, just that the library encourages some pretty bad (IMHO) behavior.

When we considered adding it before it wasn't clear that there were any features of pytest that we couldn't get just as easily from the built in unittest library. It may be that the extra complexity is justified if pytest makes writing and running tests easier. I still don't see a strong reason for using pytest in this PR, but I may be missing something here. What makes pytest --cov better than coverage run?

@@ -0,0 +1,5 @@
fixes:
- "^.*/site-packages/opentelemetry/sdk/::opentelemetry-sdk/src/opentelemetry/sdk/"

This comment has been minimized.

Copy link
@c24t

c24t Sep 11, 2019

Contributor

We may want to stop testing against site-packages to avoid this ugly hack and others like it.

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 9, 2019

Author Contributor

It's done.

@Jamim Jamim force-pushed the Jamim:feature/coverage branch 2 times, most recently from 7ffb269 to b7dea94 Oct 9, 2019
@Jamim

This comment has been minimized.

Copy link
Contributor Author

Jamim commented Oct 9, 2019

Hello everyone!

First of all, I apologize for the huge delay.

I've implemented a new approach which is based on thoughts those we discussed at the SIG meeting.
Now coverage is collected separately from the main test flow using the editable mode to install OT packages.
Could you please re-review this PR?

Thank you!

@Jamim Jamim changed the title [WIP] Add test coverage collecting Add test coverage collecting Oct 9, 2019
@Jamim Jamim force-pushed the Jamim:feature/coverage branch 2 times, most recently from 20c179f to 4999fa4 Oct 9, 2019
I believe that we should be able to see the test coverage.

These changes:

 - make tests use pytest

 - add displaying of the test coverage

 - add sending of collected coverage data to https://codecov.io
@Jamim Jamim force-pushed the Jamim:feature/coverage branch from 4999fa4 to 294ea63 Oct 9, 2019
@Oberon00

This comment has been minimized.

Copy link
Member

Oberon00 commented Oct 9, 2019

@Jamim, @hectorhdzg: So now we have both this PR and #208 active for coverage?

@Jamim

This comment has been minimized.

Copy link
Contributor Author

Jamim commented Oct 9, 2019

So now we have both this PR and #208 active for coverage?

Yes, and they are quite different in terms of the implementation.
Looks like the team should choose one of them.

I personally vote for the current one 😅

@Jamim Jamim requested review from toumorokoshi and Oberon00 Oct 9, 2019
@hectorhdzg

This comment has been minimized.

Copy link
Contributor

hectorhdzg commented Oct 9, 2019

@Oberon00 and @Jamim I closed mine in favor of this one to avoid confusion

@Jamim

This comment has been minimized.

Copy link
Contributor Author

Jamim commented Oct 9, 2019

Hello @hectorhdzg,

I understand that you had good intentions, so I have to say Thank you for pinging me and submitting that PR!

@mauriciovasquezbernal mauriciovasquezbernal referenced this pull request Oct 10, 2019
2 of 3 tasks complete
from contextvars import ContextVar
except ImportError:
pass
else:

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

why not put this in the exception?

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 11, 2019

Author Contributor

Sorry, but I don't get this question.

from . import base_context
class AsyncRuntimeContext(base_context.BaseRuntimeContext):
class Slot(base_context.BaseRuntimeContext.Slot):
def __init__(self, name: str, default: "object"):

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

couldn't the default argument just be the object object, vs the string "object"?

@@ -0,0 +1,2 @@
[pytest]
addopts = -rs -v

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

duplicate addopts?

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 11, 2019

Author Contributor

Do you mean that there is the same line inpytest-cov.ini?
That file is a separate config file used in the coverage testenv only.
It's required in order to add python_files = *.py setting which, I assume, should not be used in the test testenv.

@@ -4,6 +4,11 @@ skip_missing_interpreters = True
envlist =
py3{4,5,6,7,8}-test-{api,sdk,example-app,ext-wsgi,ext-http-requests,ext-jaeger}
pypy3-test-{api,sdk,example-app,ext-wsgi,ext-http-requests,ext-jaeger}
py3{4,5,6,7,8}-coverage

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

will this double execute all tests? Since under the hood it's calling pytest --cov, which IIUC will run all the test again.

If it is, I think we should eliminate duplicate tests before merging.

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 11, 2019

Author Contributor

will this double execute all tests?

Yes, it will.

If it is, I think we should eliminate duplicate tests before merging.

I don't think so currently.

There are reasons why:

  • As far as I know, we don't want to move back to using editable mode for the main test flow
    Please see #122 and #208/files#r332781829.
  • Non-editable mode forces us to provide fixes for Codecov.
  • There was (and probably is) a missing files issue on the Codecov side when using fixes, so it looks reasonable to use editable mode to avoid the issue.
    You can find some details in this thread.
  • Pretty ugly paths in the coverage report for non-editable mode.
  • It was suggested to make the coverage a different environment.

cc @Oberon00, @c24t

tox.ini Outdated
@@ -15,6 +15,7 @@ python =
[testenv]
deps =
mypy: mypy~=0.711
test: pytest-cov

This comment has been minimized.

Copy link
@toumorokoshi

toumorokoshi Oct 11, 2019

Collaborator

Why is editable mode required? Is that to more quickly iterate on unit tests?

I think it would be great to also benchmark the performance of the unit tests with coverage. In my experience it's not a huge overhead.

Copy link
Collaborator

toumorokoshi left a comment

I think most of my feedback is not a blocker, but I feel strongly about not double-executing tests in the CI (and hopefully not by default when I'm running tox locally).

@@ -12,32 +12,34 @@
# See the License for the specific language governing permissions and

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Oct 11, 2019

Member

I think this file was changed by mistake? Could you please remove these changes from the PR?

This comment has been minimized.

Copy link
@Jamim

Jamim Oct 11, 2019

Author Contributor

Not, it wasn't.
This file was changed in purpose to make it "importable" for Python <3.7.
Please see a corresponding build. That build failed with:

==================================== ERRORS ====================================
_ ERROR collecting opentelemetry-api/src/opentelemetry/context/async_context.py _
ImportError while importing test module '/home/travis/build/Jamim/opentelemetry-python/opentelemetry-api/src/opentelemetry/context/async_context.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
opentelemetry-api/src/opentelemetry/context/async_context.py:16: in <module>
    from contextvars import ContextVar
E   ModuleNotFoundError: No module named 'contextvars'

This comment has been minimized.

Copy link
@Oberon00

Oberon00 Oct 11, 2019

Member

But doesn't this break the logic of the importing __init__.py? You are not supposed to import that file directly anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.