Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-q no longer impacts --collect-only (when used with `--log-cli-level`) #5383

Closed
adamtheturtle opened this issue Jun 3, 2019 · 11 comments

Comments

Projects
None yet
6 participants
@adamtheturtle
Copy link

commented Jun 3, 2019

I have an application which parses pytest test collection output.

Until pytest 4.3.1 I could get easily parseable output with --collect-only -q.

~> pytest --collect-only -q tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss
tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss

no tests ran in 0.02 seconds

From @RonnyPfannschmidt here we see:

thats supported with -q for just test ids and -qq for just test filenames

However, with pytest>=4.4.0 this does not work - we see the full output even with -q:

~> pytest --collect-only -qq tests/test_dcos_e2e/backends/docker/test_distributions.py::TestCoreOS::test_oss
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.12.0 -- /Users/adam/.virtualenvs/e2e/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/adam/Documents/repositories/work/mesosphere/dcos-e2e, inifile: setup.cfg
plugins: timeout-1.3.3, cov-2.7.1, requests-mock-1.6.0
collected 1 item                                                                                                                                                                                               
<Package /Users/adam/Documents/repositories/work/mesosphere/dcos-e2e/tests/test_dcos_e2e/backends/docker>
  <Module test_distributions.py>
    Tests for the ``linux_distribution`` option on the Docker backend.
    <Class TestCoreOS>
      Tests for the CoreOS distribution option.
        <Function test_oss>
          DC/OS OSS can start up on CoreOS.

========================================================================================= no tests ran in 0.03 seconds =========================================================================================
@asottile

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

I'm not able to reproduce, could you boil this down to a minimal reproduction?

$ pytest t.py::Test  --collect-only -qq
t.py: 1


$ pytest t.py::Test  --collect-only -q
t.py::Test::test

no tests ran in 0.00 seconds
$ pytest --version
This is pytest version 4.6.2, imported from /tmp/venv/lib/python3.6/site-packages/pytest.py
@adamtheturtle

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

Sorry about that @asottile - I found that I had a tool:pytest section in my setup.cfg. I reproduced this with a new project and a minimal file - using the --log-cli-level option, set to any value:

--log-cli-level=ERROR
~/D/pytest-experiment (pytest-experiment) ~> ls
test_experiment.py
~/D/pytest-experiment (pytest-experiment) ~> cat test_experiment.py
class TestExample:
    def test_foo():
        pass
~/D/pytest-experiment (pytest-experiment) ~>
pytest --collect-only -q test_experiment.py  --log-cli-level=ERROR
========================================= test session starts ==========================================
platform darwin -- Python 3.7.3, pytest-4.6.2, py-1.8.0, pluggy-0.12.0 -- /Users/adam/.virtualenvs/pytes
t-experiment/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/adam/Desktop/pytest-experiment
plugins: timeout-1.3.3, cov-2.7.1, requests-mock-1.6.0
collected 1 item
<Module test_experiment.py>
  <Class TestExample>
      <Function test_foo>

===================================== no tests ran in 0.02 seconds =====================================
~/D/pytest-experiment (pytest-experiment) ~>

@adamtheturtle adamtheturtle changed the title -q no longer impacts --collect-only -q no longer impacts --collect-only (when used with `--log-cli-level`) Jun 4, 2019

@adamtheturtle

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

I have updated the title as appropriate.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

if self._log_cli_enabled() and config.getoption("verbose") < 1:
config.option.verbose = 1

@blueyed

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

@nicoddemus

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Thanks @adamtheturtle,

cc @thisch

@blueyed

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

@nicoddemus
You've added this initially in 8f6a592 - but previously -q would turn verbose into -1, and it was skipped.
I am not sure we should auto-enable verbose in the beginning - what was the reason for it?

@nicoddemus

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@blueyed thanks for digging this up; this was introduced in #3124 as part of the discussion in #3013.

I don't remember the specifics though. 😬

@nicoddemus

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Oh the reason for that is written in the PR description:

Live logging automatically sets the output to verbose so the logs don't get interleaved with the test progress output;

This is what we thought was a good tradeoff at the time.

@thisch

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

Since we only need this special verbose=1 code to fix the test progress output, we could also move it to a hook that is run after the collection hook. I'm currently working on a fix.

@adamtheturtle

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

Folks, I just want to say the above is really great to see - only an hour ago I narrowed this down and already there are people on it and an active discussion.

Having found the issue, I have a workaround for me - add -c '<empty file path>' to my command.

@thisch thisch referenced this issue Jun 4, 2019

Merged

Fix verbosity bug in --collect-only #5391

0 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.