Integrate mypy_primer into our CI

[intgr]

mypy_primer is a project that runs mypy on lots of public open source repositories.

In CI, it can do a mypy run before and after some proposed changes in a pull request, and report the diff between those results. It's being used by typeshed and mypy itself among others. Example:

If we add this to django-stubs and djangorestframework-stubs, it would give us better visibility into possible breakage we could be causing and help troubleshoot issues earlier.

I have no immediate plans to implement this and no experience with mypy_primer. I opened this issue to facilitate some discussion and feedback.

Most projects from mypy_primer's built-in project definitions wouldn't be very useuful for us, because they don't use Django. So we would probably want to create our own list of open source Django and django-rest-framework projects for mypy_primer. Suggestions for which ones to add would be helpful.

[sobolevn]

Related hauntsaninja/mypy_primer#8

[sobolevn]

CC @hauntsaninja

[hauntsaninja]

From the mypy_primer corpus, some potentially useful projects are: zulip, dd-trace-py, websockets, pyinstrument (from a very quick skim, probably zulip is the main thing that's useful)

[RyanWalker277]

Hey there! just wanted to chime in with some thoughts. While it's great to have better visibility into potential breakage and troubleshoot issues earlier, maintaining project-specific configurations might still be a challenge and require ongoing effort. However, if it means we can get rid of the maintenance burden of manually maintaining the list of ignores for our typecheck_tests, that would definitely be a win! Just some food for thought.

We can also consider Wagtail and Django-cms for the custom project list. Would like to hear everyone's thoughts on this.

[flaeppe]

I contributed a PR a little while back that should enable support for running mypy plugins with mypy_primer: hauntsaninja/mypy_primer#121

It hasn't been merged yet, but if we want to we could probably start by running off that PR if we want to try to get started

[intgr]

For context: typecheck_tests has been gone for ages now (#1556) and life has been much easier. Collapsing some outdated comments about that.