Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Announcement issue for plugin API changes #6617

Open
msullivan opened this issue Apr 3, 2019 · 10 comments
Open

Announcement issue for plugin API changes #6617

msullivan opened this issue Apr 3, 2019 · 10 comments

Comments

@msullivan
Copy link
Collaborator

@msullivan msullivan commented Apr 3, 2019

The mypy plugin interface is experimental, unstable, and prone to change. In particular, there are no guarantees about backwards compatibility. Backwards incompatible changes may be made without a deprecation period.

We will, however, attempt to announce breaking changes in this issue, so that plugin developers can subscribe to this issue and be notified.

Breaking changes fall into three broad categories:

  1. Changes to the actual plugin API itself
  2. Changes to parts of the "implicit plugin API"---that is, internals that plugins are likely to use (representation of types, etc).
  3. Changes to things that really aren't plausibly part of the plugin API (but, of course, that some plugins might be using anyway...)

Issues in category 1 will be consistently announced here, issues in category 3 will probably be announced here only if problems are reported, and issues in category 2 will be somewhere in the middle.

@msullivan
Copy link
Collaborator Author

@msullivan msullivan commented Apr 3, 2019

And we'll start this off with a belated category 3 announcement:

The default wheels for mypy 0.700 are compiled with mypyc. This breaks monkey-patching of mypy internals.

If you are the author of a mypy plugin that relies on monkey-patching mypy internals, get in touch with us and we can probably find a better approach. (For example, #6598 added hooks needed by the django plugin.)

Loading

msullivan added a commit that referenced this issue Apr 3, 2019
 * Link to plugin changes announcement issue (#6617)
 * Document get_additional_deps()
msullivan added a commit that referenced this issue Apr 5, 2019
 * Link to plugin changes announcement issue (#6617)
 * Document get_additional_deps()
@ethanhs ethanhs pinned this issue Apr 10, 2019
@gvanrossum gvanrossum unpinned this issue Apr 24, 2019
@gvanrossum gvanrossum pinned this issue Apr 24, 2019
@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Apr 24, 2019

(Sorry for the accidental unpin. Fixed now.)

Loading

@JelleZijlstra JelleZijlstra unpinned this issue May 3, 2019
@ilevkivskyi ilevkivskyi pinned this issue May 25, 2019
@JukkaL
Copy link
Collaborator

@JukkaL JukkaL commented Jul 11, 2019

The new semantic analyzer requires changes to some plugins, especially those that modify classes. In particular, hooks may be executed multiple times for the same definitions. PR #7135 added documentation about how to support the new semantic analyzer.

Note that mypy 0.720 (to be released soon) will enable the semantic analyzer by default, and the next release after that will remove the old semantic analyzer.

PRs #7136, #7132, #7096, #6987, #6984, #6724 and #6515 contain examples of changes that may be needed to plugins.

To test that a plugin works with the semantic analyzer, you should have test cases that cause mypy to analyze things twice. The easiest way to achieve is to add a forward reference to a type at module top level:

forwardref: C   # Forward reference to C causes deferral
class C: pass

# ... followed by whatever you want to test

Loading

@msullivan
Copy link
Collaborator Author

@msullivan msullivan commented Aug 27, 2019

PR #7397 moves around some functions as part of untangling the cyclic imports in mypy.

The most prominent change and the most likely to impact plugins is:

  • mypy.types.UnionType.make_simplified_union -> mypy.typeops.make_simplified_union

Additionally:

  • mypy.types.TypeVarDef.erase_to_union_or_bound -> mypy.types.typeops.erase_def_to_union_or_bound
  • mypy.types.TypeVarType.erase_to_union_or_bound -> mypy.types.typeops.erase_to_union_or_bound
  • mypy.types.{true_only, false_only, true_or_false} -> mypy.typeops
  • mypy.types.CallableType.corresponding_argument -> mypy.types.typeops.corresponding_argument
  • Many functions from mypy.checkmember have been moved to mypy.typeops. The most prominent here is bind_self.

Loading

@msullivan
Copy link
Collaborator Author

@msullivan msullivan commented Nov 13, 2019

PR #7829 makes all name and fullname methods in mypy into properties. This will unfortunately require changes to many plugins. We've decided that it is worth removing a long-standing pain point and that it is better to do it sooner than later.

sed can be used to update code to the new version with something like sed -i -e 's/\.name()/.name/g' -e 's/\.fullname()/.fullname/g'

If your plugin wishes to support older and newer versions during a transition period, this can be done with these helper functions:

from typing import Union
from mypy.nodes import FuncBase, SymbolNode


def fullname(x: Union[FuncBase, SymbolNode]) -> str:
    fn = x.fullname
    if callable(fn):
        return fn()
    return fn


def name(x: Union[FuncBase, SymbolNode]) -> str:
    fn = x.name
    if callable(fn):
        return fn()
    return fn

I don't have an automated way to convert code to use these, but if somebody produces one and sends it to me I will update this post.

Sorry for the inconvenience!

Loading

ilevkivskyi added a commit that referenced this issue Nov 14, 2019
This PR starts using the new `TypeAliasType` in the semantic analyzer. This PR doesn't yet pulls the trigger to enable the recursive types, but it is now essentially one line of code away. This PR:
* Makes type analyzer return a `TypeAliasType` instead of eagerly expanding the alias target.
* Refactors `TypeAliasExpr` to refer to `TypeAlias` (sorry for the noise).
* Makes few minor fixes to make all existing tests pass.
* Adds few logistic changes around `get_proper_type()` I found necessary while playing with actual recursive types over the weekend.

Here are some strategical comments:
* Taking into account how easy it was to make all existing tests pass, I don't think it is necessary to introduce a hidden option flag that would eagerly expand all type aliases after semantic analyzis.
It would probably make sense to test this well locally before a public release.
* There is a special case for no arguments generic aliases. Currently one is allowed to write `L = List; x: L[int]`, I preserve this by using eager expansion in this special case, otherwise it would complicate the whole logic significantly. This is also mostly a legacy thing because we have built-in aliases like `List = list` magically added by semantic analyzer.
* I have found that just carelessly sprinkling `get_proper_type()` is not a best strategy. It saves all the existing special-casing but also introduces a risk for infinite recursion. In particular, "type ops tangle" should ideally always pass on the original alias type. Unfortunately, there is no way to fix/enforce this (without having some severe performance impact). Note it is mostly fine to "carelessly" use `get_proper_type()` in the "front end" (like `checker.py`, `checkexpr.py`, `checkmember.py` etc).

Here is my plan for the next five PRs:
1. I am going to try merging `SubtypeVisitor` and `ProperSubtypeVisitor`, there is very large amount of code duplication (there is already an issue for this).
2. I am going to try to get rid of `sametypes.py` (I am going to open a new issue, see my arguments there).
3. I am going to enable the recursive aliases and add sufficiently many tests to be sure we are safe about infinite recursion in type ops.
4. I am going to change how named tuples and typed dicts are represented internally, currently they are stored as `TypeInfo`s, but will be stored as `TypeAlias`. Essentially there will be almost no difference between `A = Tuple[int, int]` and `A = NamedTuple('A', [('x', int), ('y', int)])`. This will allow typed dicts and named tuple participate in recursive types.
5. I am going to switch from using unbound type variables to bound type variables for generic type aliases, since now they are almost identical to `TypeInfo`s so it IMO it really makes sense to make them uniform (and avoid confusions and code duplication in future).
5a. Potentially as a follow-up I may add support for generic named tuples and typed dicts, since steps 4 plus 5 will make this almost trivial.

There is another important thing to call out, previously unions never contained another unions as items (because constructor flattened them), and some code might implicitly rely on this. IMO we should probably update these places, since maintaining this guarantee may be painful.

Yet another important thing is that this may break many plugins, so we need to announce this in #6617 when will merge this.
@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Nov 14, 2019

PR #7923 changed the internal representation of type aliases in mypy. Previously, type aliases were always eagerly expanded. For example, in this case:

Alias = List[int]
x: Alias

the type of the Var node associated with x was Instance, now it will be a TypeAliasType. This change can cause subtle bugs in plugins that make decisions using calls like if isinstance(typ, Instance): ... as such calls will now return False for type aliases.

There are two helper functions mypy.types.get_proper_type() and mypy.types.get_proper_types() that return expansions for type aliases. Note: if after making the decision on the isinstance() call you pass on the original type (and not one of its component) it is recommended to always pass on the unexpanded alias.

There is also a mypy plugin to type-check your mypy plugins, see misc/proper_plugin.py, it will flag all "dangerous" isinstance() calls.

Sorry for the inconvenience!

Loading

@ilevkivskyi
Copy link
Collaborator

@ilevkivskyi ilevkivskyi commented Nov 14, 2019

(An additional small reminder related to last two comments: don't forget that a plugin entry point gets the mypy version string, you can use it for more flexibility.)

Loading

@hauntsaninja
Copy link
Collaborator

@hauntsaninja hauntsaninja commented Aug 4, 2021

[Category 2 change] PR #9951 gets rid of TypeVarDef; use TypeVarType instead. If you're wondering what the difference between them was, so was I, which is why there's only one of them now.
cc @samuelcolvin @sobolevn @oremanj @suned @seandstewart as people who have written code that would be affected.

Loading

@sobolevn
Copy link
Contributor

@sobolevn sobolevn commented Aug 4, 2021

Loading

@JukkaL
Copy link
Collaborator

@JukkaL JukkaL commented Nov 14, 2021

#11541 causes mypy to kill the process at the end of a run, without cleaning things up properly. This might affect plugins that want to run something at the end of a run, or that assume that all files are flushed at the end of a run. --no-fast-exit can be used as a workaround, as it disables the new behavior. A better idea would be to flush files immediately.

If this change seems to cause many issues, we could consider a way of registering handlers that get run at the end of a mypy run.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
@msullivan @JukkaL @gvanrossum @sobolevn @ilevkivskyi @hauntsaninja and others