Write about experience of dropping Python 2 and using MyPy

[mrocklin]

In perusing old issues I saw two things that interested me:

1. Tornado dropped Python 2 support
2. Tornado added type annotations and MyPy type checking

I'm curious to learn more about the costs and benefits of taking these steps within a project like Tornado. I also think that I'm not alone in this curiosity. I encourage the maintainers of Tornado to write about the experience after it is done.

This isn't really an issue with Tornado, and is more a suggestion of a community activity. Please feel free to close if this is not the appropriate venue for this kind of request.

[bdarnell]

Yeah, I should probably write something up about these, especially the latter. A short version for now:

• In addition to the usual arguments in favor of Python 3 (mostly-sane str/bytes handling, even though I would have preferred the go/rust decision of utf8-everywhere), async python programs get even more benefits from being on Python 3 (better performance and cleaner syntax via native coroutines, access to the broader asyncio ecosystem). Asyncio is generating momentum around an async python ecosystem that is greater than we saw for Tornado (and, I think, Twisted, although I'm less familiar with that ecosystem). Good integration with that ecosystem is going to be crucial for the long-term health of Tornado, and it's difficult to offer that level of integration while still maintaining compatibility with python 2 and the legacy parts of the Tornado stack.

• Mypy has been more of a mixed bag. We've had bugs in the past that could have been avoided with static type checking. However, we've since built out a fairly comprehensive test suite and fixed most of these bugs. At this point I think the main benefit of adding type annotations to Tornado is to provide additional hints for code completion in IDEs, etc, rather than improvements to Tornado itself. I'm not sure yet that the investment is worth it - Jedi is pretty amazing at things like this, and mypy still has a lot of rough edges (awkward syntax, difficulties satisfying mypy and pyflakes simultaneously, inaccurate type definitions).

It's significant that I dropped Python 2 before adding mypy annotations. In theory, adding mypy annotations in Tornado 2.0 before I started adding Python 3 support (if I'm remembering my version numbers correctly) would have helped that process. In practice, I'm not sure. The type signatures of functions in the Python 2/3 hybrid versions of Tornado were very complex. I needed the simplification of dropping Python 2 (and callbacks) to make the addition of type hints tractable.

The biggest surprise in this process is that I'm not even sure that having mypy annotations would have let me avoid writing the comprehensive test suite. The test suite has been an aid to discovering the types of my functions as I attempt to write them down for mypy (although not so much the parts of the test suite that just do a bunch of type assertions). Even then I'm not sure that my type annotations are correct, which is an odd thing to say (Does this function take a List[T], a Sequence[T], or an Iterable[T]? Any of them pass the test suite, but which one matches both existing applications and future changes I might want to make?)

[mivade]

To hijack this a little bit, what do you think about adding Tornado to the list of projects here?

[bdarnell]

I think it's a good idea and mentioned it in python3statement/python3statement.github.io#21 (comment) but never got around to making a PR.

[mivade]

I could make a PR if you're OK with that.

[bdarnell]

@mivade Oops, missed this. Yes, feel free to go ahead and make a PR for the python 3 statement.

[bluetech]

Slightly off topic question:

@bdarnell, do you plan to eventually distribute Tornado's type annotations, or are they exclusively meant for internal development? By "distribute" I mean adding a py.typed marker file as described in PEP 561.

[bdarnell]

Yes, I plan to distribute the type annotations. I'm not up to date on the relevant PEPs, but it sounds like adding that py.typed marker file is the way to go. I'll be asking for feedback from users of various IDEs and toolchains when this is ready to go.

[mivade]

@bdarnell I somehow missed your missing my comment, but I've made the PR now!

[bdarnell]

Closing this since I don't think I have any more to say here than the above comment.