dir, __dir__ and __getattr__ weirdness

[fstengel]

This is in two parts

Here is a simple class

class Test(object):
    def __init__(self):
        print("Test : init")
    def __dir__(self):
        return ["Test : dir"]
    def __getattr__(self, name):
        print("Test, getattr : {}".format(name))

I would expect that using it like:

t = Test()
dir(t)

would produce:

Test : init
['Test : dir']

At least that's what the various flavours of Anaconda give me. If I try the same in micropython, I get:

Test, getattr : 
Test, getattr : __abs__
Test, getattr : __add__
...

and quite a lot of lines ending with a list of seemingly all the names known to micropython.

Now if I modify the class defined before

by removing the redefinition of __getattr__:

class Test(object):
    def __init__(self):
        print("Test : init")
    def __dir__(self):
        return ["Test : dir"]

the same test code returns:

['__class__', '__dict__', '__init__', '__module__', '__qualname__', '__dir__']

What I can deduce

Is that:

1. the __dir__ hook is not implemented (not really a problem in itself);
2. dir is very partially implemented: __getattr__ seems to be iterated over the list of all the names in the current scope. This is definitely weird.

[peterhinch]

This can be simplified further.

class Foo:
    def __getattr__(self, name):
        return 42
foo = Foo()
print(foo.bar)  # Correctly returns 42
print(len(dir(foo)))  # On my Unix MicroPython prints 687!

It is the presence of __getattr__ which provokes this behaviour.

[fstengel]

A third part

Since dict is used for tab completion, there is another issue. If one types the following:

class Foo(object):
    def __init__(self):
        self.a = []
    def __getattr__(self, name):
        return getattr(self.a, name)
 f = Foo()

then typing "f." followed by the tab key, causes this:

f.FATAL uncaught NLR xxxxxxxxx

and micropython (unix) dies nastily: the terminal in Ubuntu it ran in looses its connection to the keyboard. I have to kill it by closing the window...

[dpgeorge]

This behaviour is a consequence of the "improved" dir and tab completion implemented in #3617. It would be worth reading the commit comments in 98647e8 and 165aab1

The dir() builtin itself is only defined to give a "best effort" result, see https://docs.python.org/3/library/functions.html?highlight=dir#dir. In particular:

the function tries its best to gather information from the object’s __dict__ attribute, if defined, and from its type object. The resulting list is not necessarily complete, and may be inaccurate when the object has a custom __getattr__().

So there's an explicit caveat about undefined behaviour with __getattr__.

Also:

Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases.

MicroPython's implementation of dir attempts to be minimal and convenient, hence it probes all known names and returns those that the object responds to.

Responding to other points above:

the __dir__ hook is not implemented

Yes, right, that's an omission. And probably if you define __getattr__ you also need to define __dir__ to get any kind of sensible behaviour for the class. So this special method could be implemented.

then typing "f." followed by the tab key, causes this:

Thanks for the report about this crash, it's definitely something that needs investigation...

[dpgeorge]

The issues here have been addressed in a series of patches:

• 7241d90 fixes crashing tab-completion
• 29d28c2 improves behaviour of dir() on an object with __getattr__
• 3678a6b implements support for the __dir__ special method