Add common plugin API available for all hooks #6044

ilevkivskyi · 2018-12-09T23:33:53Z

There is a problem with get_base_class_hook(): only the full name of the base class is passed to it. This causes problems when this hook needs to interact with other hooks. In normal mode this can be solved by using a global plugin state, but this doesn't work in incremental mode, since this plugin state is not stored in cache. There is however already a field that is stored in cache between incremental runs and is free to use by plugins: TypeInfo.metadata.

The problem however is that plugin hooks get only full names and can't access the corresponding TypeInfo. This is a common enough problem that we should add an API to make the nodes referred by full names accessible to the plugin hooks. This is done by adding a common lookup method to the Plugin class.

ilevkivskyi · 2018-12-10T01:12:21Z

Btw, @mkurnikov you may be interested in this. The pattern with global state (like manager_subclasses) that you use in the Django plugin will not work in incremental mode.

JukkaL

Is the reason this is required that the generated base class is not stored in the symbol table? Otherwise, you could look up the SymbolNode for the base class using the full name, right?

Have you considered the alternative of making sure that all generated classes are included in the symbol table? I wonder if we there could be other places where we'd want to look up the TypeInfo using fullname. This wouldn't help with those.

JukkaL · 2018-12-10T13:30:58Z

test-data/unit/check-incremental.test

@@ -5586,3 +5586,75 @@ python_version=3.6
 [out]
 [out2]
 tmp/a.py:2: error: Incompatible types in assignment (expression has type "int", variable has type "str")
+
+[case testBaseClassPluginHookWorksIncremental]


I'd move this to check-custom-plugin.test and move the plugin code to a separate file.

OK, makes sense.

ilevkivskyi · 2018-12-10T14:06:34Z

Is the reason this is required that the generated base class is not stored in the symbol table?

No, the problem is that some user defined classes are special for some plugins, so the plugin can't know the set of names on which it should trigger. Previously, I tried work around this by storing all class names found to be "magic" in a plugin global state, but this doesn't work in incremental mode

ilevkivskyi · 2018-12-10T14:38:16Z

I wonder if we there could be other places where we'd want to look up the TypeInfo using fullname. This wouldn't help with those.

Hm, indeed, get_method_hook, get_method_signature_hook, get_attribute_hook, and even get_function_hook (because of generated __init__s that may be imprecise/incomplete after semantic analysis) will benefit from this (potentially also get_type_analyze_hook). But the problem with get_function_hook and get_type_analyze_hook is not clear what TypeInfo to pass there. A more global solution is to update all plugin hooks with a second argument: lookup_fully_qualified. This is however a (horribly) backwards incompatible change.

JukkaL · 2018-12-10T14:48:15Z

A more global solution is to update all plugin hooks with a second argument: lookup_fully_qualified.

Sorry, I don't understand this. What's the problem with using the api attribute? Semantic analyzer already has this, and we could add lookup_typeinfo or similar to the checker API.

ilevkivskyi · 2018-12-10T14:54:18Z

Sorry, I don't understand this. What's the problem with using the api attribute? Semantic analyzer already has this, and we could add lookup_typeinfo or similar to the checker API.

Initially we pass only full name to various get_xxx methods, and the plugin needs to decide whether to act on this name or not. A plugin can of course accept all full names, and then decide later using the full context, but then the whole idea of this first call loses all sense.

gvanrossum

Lg apart from Jukka’s suggestion.

ilevkivskyi · 2018-12-11T16:00:07Z

@JukkaL I updated the PR as we discussed. Also as you proposed, I will add the docs in a separate PR.

@msullivan Could you please check that I didn't break anything mypyc-related?

ilevkivskyi · 2018-12-11T20:05:34Z

Also @msullivan now that default plugin implementation moved out of plugin.py do we still need to compile this module? Maybe we can just leave it interpreted and remove all the related mypyc hacks?

ilevkivskyi · 2018-12-11T23:35:07Z

(Updated the title and PR description.)

msullivan · 2018-12-12T00:26:32Z

@msullivan Could you please check that I didn't break anything mypyc-related?

Seems good.

msullivan · 2018-12-12T00:27:40Z

Also @msullivan now that default plugin implementation moved out of plugin.py do we still need to compile this module? Maybe we can just leave it interpreted and remove all the related mypyc hacks?

IIRC, the main rationale for these hacks was performance. It allows the default modules to be called more efficiently than if the module interface was interpreted.

JukkaL

I think that this adds additional tech debt which we should be able to avoid pretty easily. See my comment below.

JukkaL · 2018-12-12T11:13:29Z

mypy/checker.py

@@ -3568,6 +3568,20 @@ def lookup_qualified(self, name: str) -> SymbolTableNode:
                msg = "Failed qualified lookup: '{}' (fullname = '{}')."
                raise KeyError(msg.format(last, name))

+    def lookup_fully_qualified_or_none(self, fullname: str) -> Optional[SymbolTableNode]:


I'd rather not add yet another lookup function, even if we are planning to refactor this soon. Can you move this to a new module, such as mypy.lookup? This can be a top-level function that accepts a modules dictionary and fullname. You can then pass the modules dictionary to the plugin, and the plugin can call this function (also the semantic analyzer can call the new function).

Passing the modules dictionary to plugins would have a few additional benefits: it only needs to be done once per build, and the plugins would be given less access to semantic analyzer and type checker internals.

Yes, this makes sense. I incidentally pushed before fixing some things, I will one more commit soon.

ilevkivskyi · 2018-12-12T17:38:51Z

@JukkaL OK, I updated the PR as you suggested. I took the implementation of lookup_fully_qualified() from fixup.py (it works well with both packages and nested classes), and added some comments and a docstring. Now this same function is used in fixup.py, plugin.py, and semanal.py. I didn't touch the one in checker.py because it is subtly different. Also didn't touch other lookup methods, we can do this separately.

I will update the docs PR later today.

ilevkivskyi · 2018-12-12T22:25:51Z

@msullivan

IIRC, the main rationale for these hacks was performance. It allows the default modules to be called more efficiently than if the module interface was interpreted.

Can this somehow be automated in mypyc? Updating both WrappedPlugin and InterpretedPlugin in addition to Plugin and ChainedPlugin is easy to forget.

JukkaL · 2018-12-13T10:06:40Z

mypy/plugin.py

@@ -47,6 +50,21 @@ def analyze_callable_args(self, arglist: TypeList) -> Optional[Tuple[List[Type],
        ('api', TypeAnalyzerPluginInterface)])


+@trait
+class CommonPluginApi:


Is this class needed any more?

I like to have it for documentation purpose, to separate thing that should be overridden (like get_xxx_hook() methods) from things that can be used (like options and lookup_fully_qualified()). We can however remove it if you don't like it.

JukkaL · 2018-12-13T10:08:25Z

mypy/lookup.py

+# TODO: gradually move existing lookup functions to this module.
+
+
+def lookup_fully_qualified(name: str, modules: Dict[str, MypyFile],


Can you verify that this new lookup method doesn't cause a significant slowdown using compiled mypy? The original method was simpler and if there is a performance impact, it may make sense to keep the original method as an alternative/optimization (but carefully document it as such).

I decided to keep the old look-up function in semanal.py anyway. It doesn't really belong to this PR. We can make a separate PR that cleans up all lookup related things.

ilevkivskyi · 2018-12-13T17:53:24Z

@JukkaL @gvanrossum are you happy with the PR now?

gvanrossum

LG now.

ilevkivskyi · 2018-12-13T22:31:45Z

AppVeyor build succeeded but somehow still shown as running here, I am going to merge now.

Add base class info hook

6a1e7bc

ilevkivskyi requested review from msullivan, JukkaL and gvanrossum December 9, 2018 23:33

JukkaL reviewed Dec 10, 2018

View reviewed changes

gvanrossum requested changes Dec 10, 2018

View reviewed changes

Ivan Levkivskyi added 2 commits December 11, 2018 15:44

Update based on review and discussion

dccc425

Updates for mypyc

dcf8714

ilevkivskyi changed the title ~~Add base class info hook~~ Add common plugin API available for all hooks Dec 11, 2018

ilevkivskyi mentioned this pull request Dec 11, 2018

Add plugin docs #6057

Merged

JukkaL reviewed Dec 12, 2018

View reviewed changes

Ivan Levkivskyi added 3 commits December 12, 2018 15:05

Address review comment

4148342

Soe more fixes

8e9ca53

Fix lint

eeda057

Fix self-check

23f6396

JukkaL reviewed Dec 13, 2018

View reviewed changes

Ivan Levkivskyi added 2 commits December 13, 2018 16:07

Merge remote-tracking branch 'upstream/master' into add-new-hook

490517f

More updates on review

2784af4

Undo accidental docstring change

1cfa9cd

gvanrossum approved these changes Dec 13, 2018

View reviewed changes

Remove unused file in a test

6968f55

ilevkivskyi merged commit a26b4df into python:master Dec 13, 2018

ilevkivskyi deleted the add-new-hook branch December 13, 2018 22:33

		# TODO: gradually move existing lookup functions to this module.


		def lookup_fully_qualified(name: str, modules: Dict[str, MypyFile],

Uh oh!

Add common plugin API available for all hooks #6044

Add common plugin API available for all hooks #6044

Uh oh!

Conversation

ilevkivskyi commented Dec 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi commented Dec 10, 2018

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Dec 10, 2018

Uh oh!

ilevkivskyi commented Dec 10, 2018

Uh oh!

JukkaL commented Dec 10, 2018

Uh oh!

ilevkivskyi commented Dec 10, 2018

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Dec 11, 2018

Uh oh!

ilevkivskyi commented Dec 11, 2018

Uh oh!

ilevkivskyi commented Dec 11, 2018

Uh oh!

msullivan commented Dec 12, 2018

Uh oh!

msullivan commented Dec 12, 2018

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Dec 12, 2018

Uh oh!

ilevkivskyi commented Dec 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Dec 13, 2018

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Dec 13, 2018

Uh oh!

Uh oh!

ilevkivskyi commented Dec 9, 2018 •

edited

Loading