Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"collecting" output with --color=yes in Continuous Integration output #1397

Closed
nicoddemus opened this Issue Feb 18, 2016 · 7 comments

Comments

Projects
None yet
3 participants
@nicoddemus
Copy link
Member

nicoddemus commented Feb 18, 2016

Hi,

We like to use --color=yes when running py.test in continuous integration services like Jenkins (with the Ansi Color Plugin). This enables us to see the same nice colors in Jenkins that we see on the terminal.

The only problem is that the "collecting" display progress actually piles up in the terminal because py.test output is being redirected and captured by the server. This can easily be reproduced by redirecting to a file instead:

import pytest
@pytest.mark.parametrize('i', range(10))
def test_foo(i):
    if i == 5:
        assert 0
py.test --color=yes test_foo.py > out

Contents of out:

�[1m============================= test session starts =============================�[0m
platform win32 -- Python 2.7.11, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: x:\jobs_done10, inifile: 
�[1m
collecting 0 items�[0m�[1m
collecting 10 items�[0m�[1m
collected 10 items 
�[0m
foo.py .....F....

================================== FAILURES ===================================
�[1m�[31m_________________________________ test_foo[5] _________________________________�[0m

i = 5

�[1m    @pytest.mark.parametrize('i', range(10))�[0m
�[1m    def test_foo(i):�[0m
�[1m        if i == 5:�[0m
�[1m>           assert 0�[0m
�[1m�[31mE           assert 0�[0m

foo.py:6: AssertionError
�[1m�[31m===================== 1 failed, 9 passed in 0.02 seconds ======================�[0m

The color codes are correct, but the "collecting" messages are a problem because they occupy many lines of output in test suites with hundreds of tests.

Is there any way to prevent those "collecting" messages from appearing? -q gets rid of them, but I want the header to appear, specially in CI.

@The-Compiler

This comment has been minimized.

Copy link
Member

The-Compiler commented Feb 18, 2016

This is the same as #1363, no? FWIW, I'd be okay with falling back to a static collecting... when stdout is not a terminal, independent of what --color is set to.

@nicoddemus

This comment has been minimized.

Copy link
Member Author

nicoddemus commented Feb 18, 2016

Ooops, completely forgot that I had already reported that. 😁

I agree with @The-Compiler, seems like a reasonable solution. The ideal solution proposed in #1363 would probably take a very long while to happen... just falling back to simple collecting... here would fix this problem for a ton of users without incurring too many workarounds in the code.

What do you think @RonnyPfannschmidt?

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Feb 18, 2016

I'm ambivalent, we might want to separate coloring and progress displays at some point

@nicoddemus

This comment has been minimized.

Copy link
Member Author

nicoddemus commented Feb 18, 2016

How about a --no-fancy-progress option then?

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Feb 19, 2016

We should strictly base the decision on whether the stdio is a terminal

@nicoddemus

This comment has been minimized.

Copy link
Member Author

nicoddemus commented Feb 19, 2016

Hmm I agree, I don't see any point in directing the multiple "collecting" messages anywhere else other than the terminal.

I'm ambivalent, we might want to separate coloring and progress displays at some point

I agree, but that would be a 3.0 thing right? That said, are you OK with me working on a PR for this? 😁

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Feb 20, 2016

Correct

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 20, 2016

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 20, 2016

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 20, 2016

@hpk42 hpk42 closed this in #1405 Feb 21, 2016

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Apr 14, 2016

wiz
Update py-test to 2.9.1.
2.9.1
=====

**Bug Fixes**

* Improve error message when a plugin fails to load.
  Thanks `@nicoddemus`_ for the PR.

* Fix (`#1178 <https://github.com/pytest-dev/pytest/issues/1178>`_):
  ``pytest.fail`` with non-ascii characters raises an internal pytest error.
  Thanks `@nicoddemus`_ for the PR.

* Fix (`#469`_): junit parses report.nodeid incorrectly, when params IDs
  contain ``::``. Thanks `@tomviner`_ for the PR (`#1431`_).

* Fix (`#578 <https://github.com/pytest-dev/pytest/issues/578>`_): SyntaxErrors
  containing non-ascii lines at the point of failure generated an internal
  py.test error.
  Thanks `@asottile`_ for the report and `@nicoddemus`_ for the PR.

* Fix (`#1437`_): When passing in a bytestring regex pattern to parameterize
  attempt to decode it as utf-8 ignoring errors.

* Fix (`#649`_): parametrized test nodes cannot be specified to run on the command line.


.. _#1437: pytest-dev/pytest#1437
.. _#469: pytest-dev/pytest#469
.. _#1431: pytest-dev/pytest#1431
.. _#649: pytest-dev/pytest#649

.. _@asottile: https://github.com/asottile


2.9.0
=====

**New Features**

* New ``pytest.mark.skip`` mark, which unconditionally skips marked tests.
  Thanks `@MichaelAquilina`_ for the complete PR (`#1040`_).

* ``--doctest-glob`` may now be passed multiple times in the command-line.
  Thanks `@jab`_ and `@nicoddemus`_ for the PR.

* New ``-rp`` and ``-rP`` reporting options give the summary and full output
  of passing tests, respectively. Thanks to `@codewarrior0`_ for the PR.

* ``pytest.mark.xfail`` now has a ``strict`` option, which makes ``XPASS``
  tests to fail the test suite (defaulting to ``False``). There's also a
  ``xfail_strict`` ini option that can be used to configure it project-wise.
  Thanks `@rabbbit`_ for the request and `@nicoddemus`_ for the PR (`#1355`_).

* ``Parser.addini`` now supports options of type ``bool``.
  Thanks `@nicoddemus`_ for the PR.

* New ``ALLOW_BYTES`` doctest option. This strips ``b`` prefixes from byte strings
  in doctest output (similar to ``ALLOW_UNICODE``).
  Thanks `@jaraco`_ for the request and `@nicoddemus`_ for the PR (`#1287`_).

* Give a hint on ``KeyboardInterrupt`` to use the ``--fulltrace`` option to show the errors.
  Fixes `#1366`_.
  Thanks to `@hpk42`_ for the report and `@RonnyPfannschmidt`_ for the PR.

* Catch ``IndexError`` exceptions when getting exception source location.
  Fixes a pytest internal error for dynamically generated code (fixtures and tests)
  where source lines are fake by intention.

**Changes**

* **Important**: `py.code <http://pylib.readthedocs.org/en/latest/code.html>`_ has been
  merged into the ``pytest`` repository as ``pytest._code``. This decision
  was made because ``py.code`` had very few uses outside ``pytest`` and the
  fact that it was in a different repository made it difficult to fix bugs on
  its code in a timely manner. The team hopes with this to be able to better
  refactor out and improve that code.
  This change shouldn't affect users, but it is useful to let users aware
  if they encounter any strange behavior.

  Keep in mind that the code for ``pytest._code`` is **private** and
  **experimental**, so you definitely should not import it explicitly!

  Please note that the original ``py.code`` is still available in
  `pylib <http://pylib.readthedocs.org>`_.

* ``pytest_enter_pdb`` now optionally receives the pytest config object.
  Thanks `@nicoddemus`_ for the PR.

* Removed code and documentation for Python 2.5 or lower versions,
  including removal of the obsolete ``_pytest.assertion.oldinterpret`` module.
  Thanks `@nicoddemus`_ for the PR (`#1226`_).

* Comparisons now always show up in full when ``CI`` or ``BUILD_NUMBER`` is
  found in the environment, even when ``-vv`` isn't used.
  Thanks `@The-Compiler`_ for the PR.

* ``--lf`` and ``--ff`` now support long names: ``--last-failed`` and
  ``--failed-first`` respectively.
  Thanks `@MichaelAquilina`_ for the PR.

* Added expected exceptions to ``pytest.raises`` fail message.

* Collection only displays progress ("collecting X items") when in a terminal.
  This avoids cluttering the output when using ``--color=yes`` to obtain
  colors in CI integrations systems (`#1397`_).

**Bug Fixes**

* The ``-s`` and ``-c`` options should now work under ``xdist``;
  ``Config.fromdictargs`` now represents its input much more faithfully.
  Thanks to `@bukzor`_ for the complete PR (`#680`_).

* Fix (`#1290`_): support Python 3.5's ``@`` operator in assertion rewriting.
  Thanks `@Shinkenjoe`_ for report with test case and `@tomviner`_ for the PR.

* Fix formatting utf-8 explanation messages (`#1379`_).
  Thanks `@biern`_ for the PR.

* Fix `traceback style docs`_ to describe all of the available options
  (auto/long/short/line/native/no), with `auto` being the default since v2.6.
  Thanks `@hackebrot`_ for the PR.

* Fix (`#1422`_): junit record_xml_property doesn't allow multiple records
  with same name.

.. _`traceback style docs`: https://pytest.org/latest/usage.html#modifying-python-traceback-printing

.. _#1422: pytest-dev/pytest#1422
.. _#1379: pytest-dev/pytest#1379
.. _#1366: pytest-dev/pytest#1366
.. _#1040: pytest-dev/pytest#1040
.. _#680: pytest-dev/pytest#680
.. _#1287: pytest-dev/pytest#1287
.. _#1226: pytest-dev/pytest#1226
.. _#1290: pytest-dev/pytest#1290
.. _#1355: pytest-dev/pytest#1355
.. _#1397: pytest-dev/pytest#1397
.. _@biern: https://github.com/biern
.. _@MichaelAquilina: https://github.com/MichaelAquilina
.. _@bukzor: https://github.com/bukzor
.. _@hpk42: https://github.com/hpk42
.. _@nicoddemus: https://github.com/nicoddemus
.. _@jab: https://github.com/jab
.. _@codewarrior0: https://github.com/codewarrior0
.. _@jaraco: https://github.com/jaraco
.. _@The-Compiler: https://github.com/The-Compiler
.. _@Shinkenjoe: https://github.com/Shinkenjoe
.. _@tomviner: https://github.com/tomviner
.. _@RonnyPfannschmidt: https://github.com/RonnyPfannschmidt
.. _@rabbbit: https://github.com/rabbbit
.. _@hackebrot: https://github.com/hackebrot

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue May 4, 2016

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue May 4, 2016

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue May 4, 2016

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue May 12, 2016

@joka joka referenced this issue May 30, 2016

Merged

update pytest #2363

@pyup-bot pyup-bot referenced this issue Jun 9, 2016

Merged

Initial Update #33

@pyup-bot pyup-bot referenced this issue Jul 19, 2016

Merged

Initial Update #20

@pyup-bot pyup-bot referenced this issue Aug 4, 2016

Merged

Initial Update #43

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.