[REGRESSION] -s option stops progress from being produced

[Hi-Angel]

We recently have finally migrated to newer pytest and found that the progress (controlled by console_output_style option) is no longer shown unless you make pytest capture the output (i.e. no -s option).

Last version known where it was working is 3.3.2

Steps to reproduce (in terms of terminal commands)

$ cat test1.py
def test1(): pass
def test2(): pass
def test3(): pass
$ pytest -vs ./test1.py
======================================================== test session starts ========================================================
platform linux -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /tmp/foo
collected 3 items

test1.py::test1 PASSED
test1.py::test2 PASSED
test1.py::test3 PASSED

========================================================= 3 passed in 0.00s =========================================================

Expected

On the right side there are percents shown for each test (it's the default setting of console_output_style). Like this:

[…]
test1.py::test1 PASSED                                                                                                        [ 33%]
test1.py::test2 PASSED                                                                                                        [ 66%]
test1.py::test3 PASSED                                                                                                        [100%]
[…]

Actual

Nothing is shown on the right side.

Workarounds

Replacing -s with a --capture=tee-sys seems to make it work.

Additional information

• pytest version is 7.4.2

• OS: Archlinux

• pip list

  Package               Version
  --------------------- ----------------------------------
  annotated-types       0.5.0
  appdirs               1.4.4
  async-generator       1.10
  attrs                 22.2.0
  autocommand           2.2.2
  Automat               22.10.0
  beautifulsoup4        4.12.2
  Brlapi                0.8.5
  build                 1.0.1
  CacheControl          0.13.1
  certifi               2023.7.22
  cffi                  1.15.1
  chardet               5.2.0
  click                 8.1.7
  constantly            15.1.0
  coverage              6.5.0
  crit                  3.18
  cryptography          41.0.4
  cssselect             1.2.0
  cupshelpers           1.0
  dbus-python           1.3.2
  distro                1.8.0
  docstring-to-markdown 0.12
  exceptiongroup        1.1.3
  fastjsonschema        2.18.0
  filelock              3.12.4
  h11                   0.14.0
  hyperlink             21.0.0
  idna                  3.4
  importlib-metadata    5.0.0
  incremental           22.10.0
  inflect               7.0.0
  iniconfig             2.0.0
  installer             0.7.0
  iotop                 0.6
  jaraco.context        4.3.0
  jaraco.functools      3.9.0
  jaraco.text           3.11.1
  jedi                  0.19.0
  Jinja2                3.1.2
  lensfun               0.3.4
  libfdt                1.7.0
  libmsym               0.2.4
  lockfile              0.12.2
  louis                 3.27.0
  lxml                  4.9.2
  MarkupSafe            2.1.3
  meson                 1.2.1
  more-itertools        10.1.0
  msgpack               1.0.5
  mypy                  1.3.0
  mypy-extensions       1.0.0
  netsnmp-python        1.0a1
  nftables              0.1
  numpy                 1.26.0
  ordered-set           4.1.0
  outcome               1.2.0
  packaging             23.1
  parso                 0.8.3
  perf                  0.1
  Pillow                10.0.1
  pip                   23.2.1
  platformdirs          3.10.0
  pluggy                1.3.0
  ply                   3.11
  protobuf              4.24.2
  psutil                5.9.5
  pwquality             1.4.5
  pycairo               1.24.0
  pycparser             2.21
  pycups                2.0.1
  pycurl                7.45.2
  pydantic              2.3.0
  pydantic_core         2.6.3
  PyGObject             3.46.0
  pylsp-mypy            0.6.7
  pyproject_hooks       1.0.0
  PyQt5                 5.15.9
  PyQt5-sip             12.12.2
  PyQt6                 6.5.2
  PyQt6-sip             13.5.2
  PyQt6-WebEngine       6.5.0
  PyQtWebEngine         5.15.6
  pyserial              3.5
  pytest                7.4.2
  python-lsp-jsonrpc    1.1.2
  python-lsp-server     1.8.0
  pyxdg                 0.28
  PyYAML                6.0.1
  QtPy                  2.3.1
  qutebrowser           3.0.0
  requests              2.28.2
  scour                 0.38.2
  selenium              4.12.0
  setproctitle          1.3.2
  setuptools            68.0.0
  six                   1.16.0
  smbus                 1.1
  sniffio               1.3.0
  sortedcontainers      2.4.0
  soupsieve             2.5
  TBB                   0.2
  toml                  0.10.2
  tomli                 2.0.1
  tqdm                  4.66.1
  trio                  0.22.2
  trio-websocket        0.11.1
  trove-classifiers     2023.9.22
  Twisted               22.10.0
  typing_extensions     4.8.0
  ujson                 5.8.0
  urllib3               1.26.15
  validate-pyproject    0.13.post1.dev0+gb752273.d20230520
  websockets            10.4
  wheel                 0.40.0
  wsproto               1.2.0
  yt-dlp                2023.9.24
  ytmusicapi            1.2.1
  zipp                  3.16.2
  zope.interface        6.0
  zstandard             0.21.0

[The-Compiler]

This is intended behavior. From the v3.4.2 changelog:

Removed progress information when capture option is no. (issue #3203)

[Hi-Angel]

Thanks, I see! But wouldn't it make sense to at least print a message about incompatible options being passed when both options are enabled? Because right now when I use pytest -vs -o console_output_style=progress ./test1.py the percents are just silently not being shown. And of course while I was debugging the problem I tried that as well and it didn't work, which led to the conclusion there is a bug. Some hint from the interface would save people debugging/researching time.

[Hi-Angel]

…especially so, considering the docs for console_output_style don't even mention that nuance.

[The-Compiler]

progress is the default, so this would mean we'd need to add another progress-silent-fallback. I don't think it's worth the additional complexity from an users perspective.

The docs do very much mention this:

progress-even-when-capture-no: allows the use of the progress indicator even when capture=no.

Given this line is right below progress (and progress is already the default!), I'm afraid I don't really understand how you ended up with passing -o console_output_style=progress without noticing this. Could you elaborate maybe?

[Hi-Angel]

Given this line is right below progress (and progress is already the default!), I'm afraid I don't really understand how you ended up with passing -o console_output_style=progress without noticing this. Could you elaborate maybe?

Well, yeah, I guess that's because I used the documentation a few point-releases older that google gives me when I query for console_output_style.

progress is the default, so this would mean we'd need to add another progress-silent-fallback. I don't think it's worth the additional complexity from an users perspective.

Hmm, maybe I'm missing something, I don't know pytest code… But usually you have one large block that parses all options, and you would just add in that block a bool for whether console_output_style has been explicitly modified, and then if in addition to that an -s option was passed, print an error/warning and optionally bail out.

[The-Compiler]

I'm afraid I really don't follow on the "usually" part. I can see nothing usual in behaving differently magically depending on whether an option remains on its default value, or has been set to its default value explicitly. I'd argue that should always be a no-op. It's not that it's not possible to do, but it's very surprising from an user's perspective.

[Hi-Angel]

Sorry for misunderstanding, by "usually" I was referring to the code structure. That part was a reply to your words that one would have to add a whole new separate option a la progress-silent-fallback to have the error/warning print, which kind of surprised me, because "usually code that parses options is less complicated" while admitting I don't know how it's done in pytest.

I'd argue that should always be a no-op. It's not that it's not possible to do, but it's very surprising from an user's perspective

I find it doubtful that when a user puts in two conflicting options, they really want some random behavior. "random" because a user clearly made use of the two options expecting to have specific effect for each one, and the lack of effect for either is quite random. Or to put it yet another way: would it be more logical from a user POV if -s got disabled instead of console_output_style? From a user POV there's no difference, either option being silently disabled is unexpected.

I see it's documented at your link ("loosely documented" I'd add as it only hints at the existence of such behavior), but as a user I find it pretty sad when any software has non-intuitive behavior without any guards, and instead of putting such guards just refers users to the documentation. Generally such stuff ends up being ticking-bomb bugs on the user's side (admittedly though, missing percents is not that bad of a case).