mypy not recognizing attribute of class created with setattr

[arnavb]

Let's say I have the following code in example.py:

class Example:
    def __init__(self) -> None:
        setattr(self, 'name', 'value')
    
    def a_function(self) -> bool:
        return self.name == 'value'

e = Example()

print(e.a_function())

When I run mypy example.py, I get:

example.py:6: error: "Example" has no attribute "name"

Is this expected behavior?

[gvanrossum]

Yes, mypy doesn't run your code, it only reads it, and it only looks at the types of attributes, not at actions like setattr().

[petergaultney]

I understand the reason here, but I am curious if there is any known, relatively elegant workaround for class decorators.

I am using a class decorator that adds structuring/unstructuring methods like so:

def structurer(cls):

    @staticmethod
    def structure(d: dict):
        return special_structuring_method(d, cls)

    setattr(cls, 'structure', structure)
    return cls

Obviously MyPy can't run this decorator to see what it does. But is there a relatively clean way of hinting to MyPy that specific classes do have given, named, function attributes? Obviously I could inherit from a superclass that has 'fake' versions of those methods, and that seems to be the cleanest thing I can come up with for the moment. But it would be ideal to have a method that doesn't introduce anything into the inheritance hierarchy.

I certainly understand if this is simply too dynamic for static type checking. :) It's easy to want the best of both worlds, and obviously not all things are technically feasible.

[petergaultney]

I'm looking at https://github.com/python/mypy/blob/master/mypy/plugin.py and think that a plugin using get_class_decorator_hook() might do the trick. Unfortunately I've spent the better part of the morning trying to figure out exactly how to write that plugin. If I accomplish it, I'll post what I did in case anyone else finds themselves determined to solve this problem without resorting to inheritance.

[petergaultney]

This ended up being a huge amount of work, but I was able to write a MyPy plugin that adds a static method and a regular method to the class type when it sees a specific decorator. I leaned heavily on what I found in mypy/plugins/attrs.py and mypy/plugins/common.py, but neither fully solved my problem as they did not provide actual code for adding static methods, and I had to reverse-engineer the MyPy-internal TypeInfo/ClassDefContext system.

Note that my method here is extremely use-case specific, but it's not too hard to see how this could be usefully generalized. If I can find the time, I think I will try to build out a minimal library of sorts that would allow some relatively simple syntax inside a decorator to spell out the details for a wide variety of possibilities, but there's no question that with some help from the MyPy core developers it would be a lot easier to accomplish. Perhaps someone will see this and provide suggestions for improvement.

import typing as ty
from mypy.nodes import (
    Var, Argument, ARG_POS, FuncDef, PassStmt, Block,
    SymbolTableNode, MDEF,
)
from mypy.types import NoneTyp, Type, CallableType
from mypy.typevars import fill_typevars
from mypy.semanal import set_callable_name
from mypy.plugin import Plugin, ClassDefContext
from mypy.plugins.attrs import attr_class_maker_callback
from mypy.plugins.common import add_method


class CatPlugin(Plugin):
    """A plugin to make MyPy understand Cats"""
    # TODO make struc and unstruc names configurable

    def get_class_decorator_hook(
            self,
            fullname: str
    ) -> ty.Optional[ty.Callable[[ClassDefContext], None]]:
        """One of the MyPy Plugin defined entry points"""

        def add_struc_and_unstruc_to_classdefcontext(cls_def_ctx):
            """This MyPy hook tells MyPy that struc and unstruc will be present on a Cat"""
            if fullname == 'vnxpy.cats.Cat':  # my module.decorator name
                attr_class_maker_callback(cls_def_ctx, True)  # this is only necessary because my decorator actually wraps the `attr.s` decorator, so I need it to run first.
                info = cls_def_ctx.cls.info

                if 'struc' not in info.names:
                    # here I'm basically just following a pattern from the `attrs` plugin
                    dict_type = cls_def_ctx.api.named_type('__builtins__.dict')
                    add_static_method(
                        cls_def_ctx,
                        'struc',
                        [Argument(Var('d', dict_type), dict_type, None, ARG_POS)],
                        fill_typevars(info)
                    )
                if 'unstruc' not in info.names:
                    add_method(
                        cls_def_ctx,
                        'unstruc',
                        [],
                        dict_type,
                    )
        return add_struc_and_unstruc_to_classdefcontext


def plugin(_version: str):
    """Plugin for MyPy Typechecking of Cats"""
    return CatPlugin


# i had to write this to be able to add a static method to a class type
# I only partly understand what this is actually doing.
def add_static_method(
        ctx,
        name: str,
        args: ty.List[Argument],
        return_type: Type
) -> None:
    """Mostly copied from mypy.plugins.common, with changes to make it work for a static method."""
    info = ctx.cls.info
    function_type = ctx.api.named_type('__builtins__.function')

    # args = [Argument(Var('d', dict_type), dict_type, None, ARG_POS)]
    arg_types, arg_names, arg_kinds = [], [], []
    for arg in args:
        assert arg.type_annotation, 'All arguments must be fully typed.'
        arg_types.append(arg.type_annotation)
        arg_names.append(arg.variable.name())
        arg_kinds.append(arg.kind)

    signature = CallableType(arg_types, arg_kinds, arg_names, return_type, function_type)

    func = FuncDef(name, args, Block([PassStmt()]))
    func.is_static = True
    func.info = info
    func.type = set_callable_name(signature, func)
    func._fullname = info.fullname() + '.' + name
    func.line = info.line

    info.names[name] = SymbolTableNode(MDEF, func, plugin_generated=True)
    info.defn.defs.body.append(func)

My @Cat decorator actually performs the setattrs on whatever class it decorates. It looks something like this:

def Cat(maybe_cls=None, auto_attribs=True, disallow_empties=True, **kwargs):
    def make_cat(cls):
        if not hasattr(cls, '__attrs_attrs__'):  # this part is entirely specific to my use case
            cls = cat_attrs(cls, auto_attribs=auto_attribs, **kwargs)

        @staticmethod
        def structure_dict_ignore_extras(d: dict) -> ty.Any:
            return structure_ignore_extras(d, cls)  # references a function that i want to add to my class

        setattr(cls, 'struc', structure_dict_ignore_extras)

        def unstructure_to_dict(self) -> dict:
            return unstructure(self)

        setattr(cls, 'unstruc', unstructure_to_dict)

        return cls

    if maybe_cls is not None:  # again, specific to my use case. the setattr bits above are what's relevant here.
        return make_cat(maybe_cls)
    return make_cat

[ilevkivskyi]

@petergaultney Reasonable understanding of how mypy works under the hood is a prerequisite for writing plugins (simple plugins can be written without it, but as soon as you start writing something non-trivial you will need it). After a brief look at your plugin I didn't spot any obvious bugs (apart from questionable practice to import from other plugin). You can publish this plugin on PyPI (just add a simple setup.py) if you want other people to be able to use it.

[petergaultney]

thanks, and yeah, i may publish it as part of a larger effort around typing, though the first thing to do would be to make it sufficiently generic that people could inject their own set of names, etc.

I'm wondering, however, whether I took the right approach after all. I'm in the middle of testing a theory, but perhaps you could save me some time, or let me know about a minefield that I'm missing. It seems, now that I take another look at the classes in mypy/nodes.py, that the easy way to accomplish what I did would be to instead write a simple 'model' class with the code containing the types that I wanted to add, run mypy over that and have it generate the FuncDefs (or even attribute nodes, as in the original question) for me, serialize those FuncDefs, and then, in my plugin, deserialize the model methods, inject the appropriate names, and give those back to MyPy. Does this actually work, or am I moving into dangerous territory here?

If it did work, it seems to me that it would not be too difficult to write a "decorator meta-typing plugin", that could effectively be pointed at static code that "shows" the methods, attributes, etc. that a decorator is intended to apply to a class, and applies those wherever it sees the decorator being used.

[petergaultney]

Looks like deserializing a serialized SymbolTableNode or a FuncDef gets me an assertion error "De-serialization failure: TypeInfo not fixed." So this isn't quite as straightforward as I hoped. Is there something I can do to make a SymbolTableNode or FuncDef properly deserialize by fixing up the "TypeInfo" afterwards, or is this going to be difficult to do as dynamically as I was hoping?

e.g.:
{'.class': 'SymbolTableNode', 'kind': 'Mdef', 'node': {'.class': 'FuncDef', 'name': 'a_normal_method', 'fullname': 'cereal.TestCereal.a_normal_method', 'arg_names': ['self', 'mystr', 'mydict'], 'arg_kinds': [0, 0, 0], 'type': {'.class': 'CallableType', 'arg_types': ['cereal.TestCereal', 'builtins.str', 'builtins.dict'], 'arg_kinds': [0, 0, 0], 'arg_names': ['self', 'mystr', 'mydict'], 'ret_type': 'builtins.dict', 'fallback': 'builtins.function', 'name': 'a_normal_method of TestCereal', 'variables': [], 'is_ellipsis_args': False, 'implicit': False, 'bound_args': [], 'def_extras': {'first_arg': 'self'}}, 'flags': []}} is the serialized version, but when I do SymbolTableNode.deserialize() on it, I end up getting an error as soon as I try to do anything with it.

[ilevkivskyi]

If it did work, it seems to me that it would not be too difficult to write a "decorator meta-typing plugin", that could effectively be pointed at static code that "shows" the methods, attributes, etc. that a decorator is intended to apply to a class, and applies those wherever it sees the decorator being used.

De-serializing mypy cache is not a safe idea. First of all the cache format is totally a private API and will likely change. Second, mypy supports (mutually) recursive classes, so de-serialization requires an additional fix-up pass, which is no-trivial. You can alternatively try getting the actual AST objects (all things can be found from plugins by their full names), but then you will need to somehow copy them. Although the final goal (write a generic plugin that understands meta-programming source code) looks interesting, this is a non-trivial task and requires deep understanding of how mypy works.

[petergaultney]

I really appreciate the feedback! I didn't realize the serialization was primarily intended for the cache.

I'll keep thinking about it. :)

[huyz]

@petergaultney did you get around to doing more work on this or publishing it?

[petergaultney]

I did a little more on it and abandoned it for lack of time. It still seems like it should be theoretically possible to build some sort of helper that would allow plugins to be written dynamically, rather than piece-by-piece, since at the end of the day you're just trying to teach mypy that certain "names" match certain types that are otherwise as easy to write as def foo(x: str) -> int.