Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage by using __slots__ #4904

Merged
merged 11 commits into from Apr 26, 2018

Conversation

Projects
None yet
2 participants
@JukkaL
Copy link
Collaborator

JukkaL commented Apr 13, 2018

In some cases this reduces the memory usage of mypy and mypy-daemon
over 30%. This seems to have only a minor effect on speed. Depending on
the workload, this seems to either speed up or slow down runtimes by a
few percent.

Notes:

  • Refactor various attribute initializations away from class bodies, as they
    conflict with __slots__.
  • This only adds __slots__ to classes that use the most memory, based
    on the output from the memory profiler.
  • Some attributes had None defaults even though the type was not
    optional. Now we use a cast for some of these, which has the same
    effect.
  • Various __dict__ manipulations had to be modified to support
    __slots__ as well.
@gvanrossum
Copy link
Member

gvanrossum left a comment

Oops, I think I forgot to send these comments.

can_be_false = False
__slots__ = ('fallback',)

MYPY = False # Use this to declare, we don't want a runtime None value

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Hm, I'd rather see MYPY = False as a global (though the comment belongs in the class).

MYPY = False # Use this to declare, we don't want a runtime None value
if MYPY:
# Corresponding instance type (e.g. builtins.type)
fallback = None # type: Instance

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Why not do this in __init__()?

"""Type of a non-overloaded callable object (such as function).
Attributes:
arg_types: Types of function arguments

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

(What a bummer we can't use PEP 526 notation yet. :-( )

'implicit',
'special_sig',
'from_type_type',
'bound_args')

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

I'd prefer a trailing comma here. Also perhaps you can add the comments here rather than to the docstring?


def replace_object_state(new: object, old: object) -> None:
# Copy state of old node to the new node. This varies depending on whether
# there is __slots__ and/or __dict__.

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

I feel this should be a docstring, not a comment.

if hasattr(old, '__dict__'):
new.__dict__ = old.__dict__
if hasattr(old, '__slots__'):
# Use type.mro(...) since some classes override 'mro' with something different.

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Or use __mro__, I doubt they override that.

new.__dict__ = old.__dict__
if hasattr(old, '__slots__'):
# Use type.mro(...) since some classes override 'mro' with something different.
for base in type.mro(type(old)):

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

I wonder if it would be worth it caching the combined set of slots on the class, so you don't have to walk the MRO each time.

def __init__(self) -> None:
super().__init__()
# Type signature. This is usually CallableType or Overloaded, but it can be
# something else for decorated functions/

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Fix the trailing /

'is_awaitable_coroutine',
'is_static',
'is_class',
'expanded')

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Add trailing comma (and move ) to next line, also in other classes below), and consider adding the comments here rather than to the docstring?

@@ -507,22 +558,23 @@ def is_dynamic(self) -> bool:
return self.type is None


FUNCDEF_FLAGS = FUNCITEM_FLAGS + [
'is_decorated', 'is_conditional', 'is_abstract', 'is_property'

This comment has been minimized.

@gvanrossum

gvanrossum Apr 15, 2018

Member

Trailing comma.

@JukkaL

This comment has been minimized.

Copy link
Collaborator Author

JukkaL commented Apr 24, 2018

I think that I've addressed all comments (or at least most).

@gvanrossum
Copy link
Member

gvanrossum left a comment

All looks good now! You have a merge conflict though. :-(

@JukkaL JukkaL merged commit 498a9cc into master Apr 26, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@gvanrossum gvanrossum deleted the memory-opt branch Apr 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.