Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize dataclasses with __slots__ #50

Open
nat-n opened this issue May 21, 2020 · 8 comments
Open

Optimize dataclasses with __slots__ #50

nat-n opened this issue May 21, 2020 · 8 comments
Labels
enhancement New feature or request low priority medium Medium effort issue, can fit in a single PR
Projects

Comments

@nat-n
Copy link
Collaborator

nat-n commented May 21, 2020

Adding __slots__ to a class improves memory usage and performance of attribute access, at the expense of not being able to set arbitrary attributes on the resulting objects.

This project demonstrates how dataclasses can be augmented to use slots, whilst still having class attributes set (normally adding the slots attribute to a class definition would cause an error with dataclassess that declare default values).

Making generated dataclasses use slots in this way would be a small breaking change, though I don't see how it could be too inconvenient to work with, and I expect the performance improvement would be worthwhile.

@nat-n nat-n added the enhancement New feature or request label May 21, 2020
@boukeversteegh boukeversteegh added this to Backlog in Betterproto May 25, 2020
@boukeversteegh boukeversteegh added the medium Medium effort issue, can fit in a single PR label May 25, 2020
@boukeversteegh boukeversteegh moved this from Backlog to Icebox in Betterproto May 25, 2020
@Gobot1234
Copy link
Collaborator

Gonna give this one a shot 👍

@Gobot1234
Copy link
Collaborator

>>> @dataclass
... class Data:
...     t: int
...     m: str
...     n: float
... 
>>> Data(t=3, m=3, n=3).__sizeof__()
32
>>> @dataslots.with_slots
... @dataclass
... class Data:
...     t: int
...     m: str
...     n: float
... 
>>> Data(t=3, m=3, n=3).__sizeof__()
40

Really not seeing much in the ways of benefits here.

Attribute access is marginally faster as is instantiation but this doesn't seem like a worthwhile trade off against the extra overhead and memory size.

I might be seriously missing something or it might be Python 3.9 doing stuff not too sure though.

@nat-n
Copy link
Collaborator Author

nat-n commented Aug 25, 2020

I'm curious did you find a measurable speed improvement?
Is the memory penalty constant or proportional to the number of fields? (or does it turn to a benefit with enough fields?)

Admittedly I've not done enough investigation to validate this, but my thinking was the if the technique from dataslots were implemented directly in the Message class then it should give the most benefit.

@Gobot1234
Copy link
Collaborator

The there didn't appear to be a point at which the un-slotted class used more memory. The speed improvement wasn't noteworthy. Might have to test some more however.

@Gobot1234
Copy link
Collaborator

Actually I take this back it appears to be working now 🤔

@Gobot1234
Copy link
Collaborator

All tests were ran with timeit.repeat and a sped up custom implementation of dataslots.

So for a single slotted class attribute access 0.11287586920000003s vs unslotted 0.1498976172s

Init appears to be slightly slower slotted 25.6038842112s vs 22.337924385599997s.

Overhead is relatively low at 3.2458480566e-05s per class per decorator directly using the implementation of the above library finishes in 3.7460388750999996e-05s

Memory usage temperamental which I think is caused by the use of betterproto.int32_field(0) etc. So I think it's safe to say the previous statements still apply,

@Gobot1234
Copy link
Collaborator

The size issues might be due to using it on instances this sure is very strange.

@Gobot1234
Copy link
Collaborator

I'm gonna give a Message metaclass a go for the best performance as you suggested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request low priority medium Medium effort issue, can fit in a single PR
Projects
Betterproto
  
Icebox
Development

Successfully merging a pull request may close this issue.

3 participants