Failure still reported when whole module gets re-run

[blueyed]

I have just seen this, after marking TestSchemaJSRenderer (which only
contains test_schemajs_output) with
@pytest.mark.skipif(not coreapi, reason='coreapi is not installed'):

% .tox/venvs/py36-django20/bin/pytest --testmon
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
testmon=True, changed files: tests/test_renderers.py, skipping collection of 118 files, run variant: default
rootdir: /home/daniel/Vcs/django-rest-framework, inifile: setup.cfg
plugins: testmon-0.9.11, django-3.2.1, cov-2.5.1
collected 75 items / 1219 deselected                                                                                                                                                         

tests/test_renderers.py F                                                                                                                                                              [  0%]
tests/test_filters.py F                                                                                                                                                                [  0%]
tests/test_renderers.py ..............................................Fs                                                                                                               [100%]

========================================================================================== FAILURES ==========================================================================================
_________________________________________________________________________ TestSchemaJSRenderer.test_schemajs_output __________________________________________________________________________
tests/test_renderers.py:753: in test_schemajs_output
    output = renderer.render('data', renderer_context={"request": request})
rest_framework/renderers.py:862: in render
    codec = coreapi.codecs.CoreJSONCodec()
E   AttributeError: 'NoneType' object has no attribute 'codecs'
_________________________________________________________________ BaseFilterTests.test_get_schema_fields_checks_for_coreapi __________________________________________________________________
tests/test_filters.py:36: in test_get_schema_fields_checks_for_coreapi
    assert self.filter_backend.get_schema_fields({}) == []
rest_framework/filters.py:36: in get_schema_fields
    assert coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`'
E   AssertionError: coreschema must be installed to use `get_schema_fields()`
________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'
====================================================================================== warnings summary ======================================================================================
None
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.

-- Docs: http://doc.pytest.org/en/latest/warnings.html
======================================================== 3 failed, 46 passed, 1 skipped, 1219 deselected, 1 warnings in 3.15 seconds =========================================================

TestSchemaJSRenderer.test_schemajs_output should not show up in FAILURES.

Note also that tests/test_renderers is listed twice, where the failure
appears to come from the first entry.

(running without --testmon shows the same number of tests (tests/test_renderers.py ..............................................Fs)).

[blueyed]

Changing it back and forth running .tox/venvs/py36-django20/bin/pytest tests/test_renderers.py --testmon gave me the correct output, with colored (i.e. fresh output):

tests/test_renderers.py ..............................................Fs

Changing it to be 2 failures then again, and running it without the path then
however re-ran most of the tests, only de-selecting the 48 tests from
tests/test_renderers.py:

collected 1146 items / 48 deselected
tests/test_renderers.py ..............................................FF

FAILURES

________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'
_________________________________________________________________ BaseFilterTests.test_get_schema_fields_checks_for_coreapi __________________________________________________________________
tests/test_filters.py:36: in test_get_schema_fields_checks_for_coreapi
    assert self.filter_backend.get_schema_fields({}) == []
rest_framework/filters.py:36: in get_schema_fields
    assert coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`'
E   AssertionError: coreschema must be installed to use `get_schema_fields()`
________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'
_________________________________________________________________________ TestSchemaJSRenderer.test_schemajs_output __________________________________________________________________________
tests/test_renderers.py:753: in test_schemajs_output
    output = renderer.render('data', renderer_context={"request": request})
rest_framework/renderers.py:862: in render
    codec = coreapi.codecs.CoreJSONCodec()
E   AttributeError: 'NoneType' object has no attribute 'codecs'

Note that the first and 3 are the same.
The first is not colored (i.e. cached), but the module was re-run and it was reported again.

Changing it to be skipped again then even resulted in this:

tests/test_renderers.py FFF                                                                                                                                                            [  0%]
tests/test_filters.py F                                                                                                                                                                [  0%]
tests/test_authentication.py .......................................................                                                                                                   [ 48%]
tests/test_authtoken.py ..........                                                                                                                                                     [ 57%]
tests/test_renderers.py ..............................................Fs                                                                                                               [100%]

FAILURES

tests/test_renderers.py:753: in test_schemajs_output
    output = renderer.render('data', renderer_context={"request": request})
rest_framework/renderers.py:862: in render
    codec = coreapi.codecs.CoreJSONCodec()
E   AttributeError: 'NoneType' object has no attribute 'codecs'
________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'
________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'
_________________________________________________________________ BaseFilterTests.test_get_schema_fields_checks_for_coreapi __________________________________________________________________
tests/test_filters.py:36: in test_get_schema_fields_checks_for_coreapi
    assert self.filter_backend.get_schema_fields({}) == []
rest_framework/filters.py:36: in get_schema_fields
    assert coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`'
E   AssertionError: coreschema must be installed to use `get_schema_fields()`
________________________________________________________________ TestDocumentationRenderer.test_document_with_link_named_data ________________________________________________________________
tests/test_renderers.py:719: in test_document_with_link_named_data
    document = coreapi.Document(
E   AttributeError: 'NoneType' object has no attribute 'Document'

(1-4 are cached/not colored, 5 is fresh and the same as 2 and 3).

[blueyed]

It is a unittest based test btw: https://github.com/blueyed/django-rest-framework/blob/d77784f8d27b9047e20992a5e23556ec4bb79b11/tests/test_renderers.py#L741

[blueyed]

Not sure if #102 fixes everything from here, but at least some.

[tarpas]

I spend some time on this, but don't understand neither the bug report nor the pull request.

Is there simple sequence of steps which reproduce the bug?

What does the title mean? How does one re-run whole module? What commands did you use?

I'm a basic user of tox, so maybe I'm missing some context which you might expect I'm familiar with but I'm not.

[tarpas]

I have a new hypothesis of what you mean, so i'll investigate that.

[tarpas]

@blueyed Daniel, thanks for reporting. I'm aware of the instability of testmon at the moment and I'm sorry for that. As said earlier your bug report is not clear enough to allow me to move forward with it. I hope to get somebody to help me with testmon but that takes some time .. I'll keep you informed but it might 2 or so weeks if everything goes great.

[blueyed]

@tarpas
"The whole module gets re-run" happens since I am adding (or removing) a decorator on one of the test classes.
testmon then first reports the known failure, but later re-runs all the tests (including the already reported one).
The PR removes the known failure from the tests that should get reported (since it will be re-run).

I'm aware of the instability of testmon at the moment and I'm sorry for that

Don't worry - it is still very helpful and a great thing to have in general!

I hope to get somebody to help me with testmon but that takes some time

What is your plan in this regard?

[blueyed]

Fixed on master now, thanks a lot!