Reactor unclear on a twisted unit test when using content()

[gsemet]

Hello

I am on twisted 15 on windows with latest version of treq and I have the following issue:

• when, in a unit test, I execute treq.content(response), everything work perfecly until the end of the trial unit test. reponse is the response of treq.get(). It issues a Reactor unclear exception (delayed response). do you have seen such issue? Only occurs when retrieving the content, if I don't do the treq.content(), there is not problem with the reactor.

thanks

[hynek]

treq maintains a connection pool that you will have to close by hand like in our tests: https://github.com/dreid/treq/blob/master/treq/test/test_treq_integration.py#L63

[gsemet]

Can you update the doc accordingly?

[hynek]

yeah, I’ve opened a bug for that, thanks.

[gsemet]

Ok thanks a lot

[posita]

UPDATE: This was a silly oversight on my part. I forgot to return the Deferred from each of my test methods!

Please disregard my original comment, which is preserved below.

I am still seeing these despite having a cleanup method similar to yours. test_treq_json.py:

#!/usr/bin/env python
#-*-mode: python; encoding: utf-8-*-

from __future__ import (
    absolute_import,
    division,
    print_function,
    unicode_literals,
    with_statement,
)

#---- Imports ------------------------------------------------------------

from unittest import main as unittest_main
from twisted.internet import reactor
from twisted.internet.task import deferLater
from twisted.internet.tcp import Client
from twisted.trial.unittest import TestCase
from twisted.web.client import HTTPConnectionPool
from treq import (
    get as treq_get,
    json_content as treq_json_content,
)

#---- Constants ----------------------------------------------------------

__all__ = ()

_GOOD_URL = b'https://blockchain.info/rawblock/0000000000000bae09a7a393a8acded75aa67e46cb81f7acaa5ad94f9eacd103'
_FAIL_URL = b'https://doesnotexist.dom/not/a/real/path'

#---- Classes ------------------------------------------------------------

#=========================================================================
class TestJsonRest(TestCase):

    #---- Public hook methods --------------------------------------------

    #=====================================================================
    def setUp(self):
        # See <https://github.com/twisted/treq/issues/87>
        self.pool = HTTPConnectionPool(reactor, False)

    #=====================================================================
    def tearDown(self):
        # See <https://github.com/twisted/treq/issues/87>
        def _check_fds(_):
            fds = set(reactor.getReaders() + reactor.getReaders())

            if not [ fd for fd in fds if isinstance(fd, Client) ]:
                return

            return deferLater(reactor, 0, _check_fds, None)

        return self.pool.closeCachedConnections().addBoth(_check_fds)

    #=====================================================================
    def test_get(self):
        d = treq_get(_GOOD_URL, pool=self.pool)
        d.addCallback(treq_json_content)

        def _check_response(json):
            pass

        d.addCallback(_check_response)

    #=====================================================================
    def test_get_fail(self):
        d = treq_get(_FAIL_URL, pool=self.pool)
        d.addCallback(treq_json_content)

        def _check_response(json):
            pass

        d.addCallback(_check_response)

#---- Initialization -----------------------------------------------------

if __name__ == '__main__':
    unittest_main()

Results:

% python tests/test_treq_json.py
.EE
======================================================================
ERROR: test_get (__main__.TestJsonRest)
test_get
----------------------------------------------------------------------
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x10de46ac0 [29.9983620644s] called=0 cancelled=0 Client.failIfNotConnected(TimeoutError('',))>
<DelayedCall 0x10de46ae0 [59.9997429848s] called=0 cancelled=0 ThreadedResolver._cleanup('blockchain.info', <Deferred at 0x10de46aa0>)>

======================================================================
ERROR: test_get_fail (__main__.TestJsonRest)
test_get_fail
----------------------------------------------------------------------
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x10e0f4d80 [29.9989640713s] called=0 cancelled=0 Client.failIfNotConnected(TimeoutError('',))>
<DelayedCall 0x10e0f4da0 [59.9998269081s] called=0 cancelled=0 ThreadedResolver._cleanup('doesnotexist.dom', <Deferred at 0x10e0f4d60>)>

----------------------------------------------------------------------
Ran 3 tests in 0.021s

FAILED (errors=2)

Any ideas? 😕