reduce uuid.UUID() memory footprint

[wbolster]

BPO	30977
Nosy	@vstinner, @taleinat, @serhiy-storchaka, @wbolster
PRs	bpo-30977: define uuid.UUID.__slots__ to save memory #2785 bpo-30977: make uuid.UUID use __slots__ to reduce its memory footprint #9078 bpo-30977: rework code changes according to post-merge code review #9106 [3.7] bpo-34621: fix uuid.UUID (un)pickling compatbility with older Python versions (<3.7) #9133

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-06.11:35:57.364>
created_at = <Date 2017-07-20.17:15:28.449>
labels = ['3.8', 'library', 'performance']
title = 'reduce uuid.UUID() memory footprint'
updated_at = <Date 2018-09-10.13:11:09.040>
user = 'https://github.com/wbolster'

bugs.python.org fields:

activity = <Date 2018-09-10.13:11:09.040>
actor = 'taleinat'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-06.11:35:57.364>
closer = 'taleinat'
components = ['Library (Lib)']
creation = <Date 2017-07-20.17:15:28.449>
creator = 'wbolster'
dependencies = []
files = []
hgrepos = []
issue_num = 30977
keywords = ['patch']
message_count = 11.0
messages = ['298738', '298739', '298757', '298785', '303194', '324665', '324671', '324672', '324682', '324683', '324923']
nosy_count = 5.0
nosy_names = ['vstinner', 'taleinat', 'serhiy.storchaka', 'wbolster', 'Nir Soffer']
pr_nums = ['2785', '9078', '9106', '9133']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue30977'
versions = ['Python 3.8']

[wbolster]

memory usage for uuid.UUID seems larger than it has to be. it seems that using __slots__ will save around ~100 bytes per instance, which is very significant, e.g. when dealing with large sets of uuids (which are often used as "primary keys" into external data stores).

uuid.UUID has a __setattr__ that prevents any extra attributes to be
set:

    def __setattr__(self, name, value):
        raise TypeError('UUID objects are immutable')

...so it seems to me not having __dict__ should not cause any problems?

before (RES column):

>> import uuid
>> y = {uuid.uuid4() for _ in range(1000000)}

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
23020 wbolster 20 0 315M 253M 7256 S 0.0 1.6 0:04.98 python

with slots:

>> import uuid
>> y = {uuid.uuid4() for _ in range(1000000)}

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
21722 wbolster 20 0 206M 145M 7348 S 0.0 0.9 0:05.03 python

i will open a pr on github shortly.

[wbolster]

as a follow-up note, i also experimented with keeping the actual value as a bytes object instead of an integer, but that does not lead to less memory being used: a 128-bit integer uses less memory than a 16 byte bytes object (presumably because PyBytesObject has a cached hash() field and a trailing null byte).

[NirSoffer]

This saves memory, but using str(uuid.uuid4()) requires even less memory.
If you really want to save memory, you can keep the uuid.uuid4().int.

Can you explain someone would like to have 1000000 uuid objects, instead of 1000000 strings? What is the advantage of keeping UUID objects around?

[wbolster]

i consider uuids as low level data types, not as fancy containers, similar to how i view datetime objects. given the native support in e.g. postgresql and many other systems, it's common to deal with uuids.

of course you can convert to/from strings or numbers, but that is cumbersome in many cases. for comparison, one would typically not convert unicode text from/into utf-8 encoded byte strings either, even though the latter will save memory in many cases.

from experience: converting can lead to nasty bugs, e.g. because you forgot about a conversion, and then a uuid string does not compare equal to a uuid.UUID instance, leaving you puzzled.

[serhiy-storchaka]

This change breaks pickle.

You should preserve forward and backward pickle compatibility.

1. Pickle data produced by old Python versions should be unpickleable with a new implementation. Implement __setstate__ for satisfying this.

2. Pickle data produced by a new implementation should be unpickleable in old Python versions. There are many ways to satisfy this, you should choose the most efficient.

[taleinat]

See new PR which addresses pickle forward and backward compatibility.

[vstinner]

I close the issue because of the pickle issue that hasn't been addressed by the wouter bolsterlee (the author) didn't reply for 1 month 1/2.

@wouter bolsterlee: if you still want to work on that issue, you should try to address the pickle issue first, then reopen this issue or maybe create a new issue pointing to this one.

[vstinner]

Oops. I missed the fact that Tal created PR 9078. Sorry, I reopen the issue ;-)

[taleinat]

New changeset 3e2b29d by Tal Einat in branch 'master':
bpo-30977: make uuid.UUID use __slots__ (GH-9078)
3e2b29d

[taleinat]

Thanks for the suggestion and the original patch, Wouter!

[taleinat]

New changeset 5475253 by Tal Einat in branch 'master':
bpo-30977: rework code changes according to post-merge code review (GH-9106)
5475253