Test with Tornado 5.0

[bdarnell]

(I know y'all are on top of this, but I'll go ahead and file this here for final confirmation)

Tornado 5.0 is currently in beta, and I'm filing issues with prominent
users of Tornado to give them a chance to test it before release and
provide feedback on the changes and make adjustments as necessary. You
can install it with pip install --pre tornado==5.0b1. I expect to
release the final version of Tornado 5.0 in early March, about a month
away.

Full release notes are
here. The
major change is that the IOLoop is now always backed by asyncio on
Python 3, which leads to various subtle changes at runtime but is
mostly compatible with previous versions. A more superficial but
source-incompatible change is that the io_loop argument to many
functions in Tornado has been removed and IOLoop.current() must be
used instead.

If you've already tested with Tornado 5.0b1 (or temporarily pinned
your dependency to 4.x so you can postpone dealing with this), feel
free to close this issue.

[mrocklin]

I'm very glad to see a beta out. I look forward to checking the current status of things. After the release we may also want to write about the bandwidth improvements. This would be, I think, of interest to the scientific and HPC communities.

…

On Tue, Jan 30, 2018 at 8:08 PM, Ben Darnell ***@***.***> wrote: (I know y'all are on top of this, but I'll go ahead and file this here for final confirmation) Tornado 5.0 is currently in beta, and I'm filing issues with prominent users of Tornado to give them a chance to test it before release and provide feedback on the changes and make adjustments as necessary. You can install it with pip install --pre tornado==5.0b1. I expect to release the final version of Tornado 5.0 in early March, about a month away. Full release notes are here <http://www.tornadoweb.org/en/latest/releases/v5.0.0.html>. The major change is that the IOLoop is now always backed by asyncio on Python 3, which leads to various subtle changes at runtime but is mostly compatible with previous versions. A more superficial but source-incompatible change is that the io_loop argument to many functions in Tornado has been removed and IOLoop.current() must be used instead. If you've already tested with Tornado 5.0b1 (or temporarily pinned your dependency to 4.x so you can postpone dealing with this), feel free to close this issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1725>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszMnGZaK899597WkfMyA2JsSn7cWcks5tP70GgaJpZM4RzRJn> .

[mrocklin]

It looks like we do have some work to do. The test suite doesn't fully pass. cc @pitrou in case he's interested (though I suspect he is occupied with other things). On Tue, Jan 30, 2018 at 9:01 PM, Matthew Rocklin <mrocklin@anaconda.com> wrote:

…

I'm very glad to see a beta out. I look forward to checking the current status of things. After the release we may also want to write about the bandwidth improvements. This would be, I think, of interest to the scientific and HPC communities. On Tue, Jan 30, 2018 at 8:08 PM, Ben Darnell ***@***.***> wrote: > (I know y'all are on top of this, but I'll go ahead and file this here > for final confirmation) > > Tornado 5.0 is currently in beta, and I'm filing issues with prominent > users of Tornado to give them a chance to test it before release and > provide feedback on the changes and make adjustments as necessary. You > can install it with pip install --pre tornado==5.0b1. I expect to > release the final version of Tornado 5.0 in early March, about a month > away. > > Full release notes are > here <http://www.tornadoweb.org/en/latest/releases/v5.0.0.html>. The > major change is that the IOLoop is now always backed by asyncio on > Python 3, which leads to various subtle changes at runtime but is > mostly compatible with previous versions. A more superficial but > source-incompatible change is that the io_loop argument to many > functions in Tornado has been removed and IOLoop.current() must be > used instead. > > If you've already tested with Tornado 5.0b1 (or temporarily pinned > your dependency to 4.x so you can postpone dealing with this), feel > free to close this issue. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1725>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszMnGZaK899597WkfMyA2JsSn7cWcks5tP70GgaJpZM4RzRJn> > . >

[pitrou]

Yeah, a bunch of tests are timing out... I don't have the time or motivation to diagnose this, sorry ;-)

[mrocklin]

I don't have the time or motivation to diagnose this, sorry ;-)

No worries at all

[mrocklin]

I have most things working here:

There were a couple changes to note. First, our use of Tornado 5.0 seems to not clean up file descriptors as well as it did with Tornado 4.5. They do seem to eventually get cleaned up, but it takes a while.

@pytest.mark.skipif(sys.platform.startswith('win'),
                    reason="num_fds not supported on windows")
def test_client_num_fds(loop):
    psutil = pytest.importorskip('psutil')
    with cluster() as (s, [a, b]):
        proc = psutil.Process()
        before = proc.num_fds()
        with Client(s['address'], loop=loop) as c:
            during = proc.num_fds()
        after = proc.num_fds()

        assert before >= after

[mrocklin]

Whoops, forgot to link to the PR: #1735

[bdarnell]

First, our use of Tornado 5.0 seems to not clean up file descriptors as well as it did with Tornado 4.5. They do seem to eventually get cleaned up, but it takes a while.

This was tricky for Tornado's own tests too. There is always a background asyncio event loop (Tornado restores the original one whenever you call clear_current), and it has at least 3 fds open at all times. Failure to call clear_current (in between tests, for example) can leak this event loop and leave its fds open until the GC runs.

[mrocklin]

Your comment helped me to resolve the issue. Thanks!

[bdarnell]

I see this issue is still open. Is everything ready on your side? I'm planning to release Tornado 5.0 in the next week.

[mrocklin]

I believe that everything is good. Have there been changes since 5.0b1 ? If so then is there a particular tag or commit that I should test against?

[bdarnell]

There have been a few minor bug fixes but nothing that might cause any problems. I'm planning to cut the release from the master branch.