Custom pickler memory leak

[douglas-raillard-arm]

BPO	45830
Nosy	@douglas-raillard-arm

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-17.16:42:28.795>
labels = ['library', '3.10', 'performance']
title = 'Custom pickler memory leak'
updated_at = <Date 2021-11-17.16:42:28.795>
user = 'https://github.com/douglas-raillard-arm'

bugs.python.org fields:

activity = <Date 2021-11-17.16:42:28.795>
actor = 'douglas-raillard-arm'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-11-17.16:42:28.795>
creator = 'douglas-raillard-arm'
dependencies = []
files = []
hgrepos = []
issue_num = 45830
keywords = []
message_count = 1.0
messages = ['406478']
nosy_count = 1.0
nosy_names = ['douglas-raillard-arm']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue45830'
versions = ['Python 3.10']

[douglas-raillard-arm]

The following script exhibits the memory leak. It only happens if "dispatch_table" is set _before_ calling super().__init__, which is pretty unexpected.

    import pickle
    import io
    import gc
    import tracemalloc

    tracemalloc.start(10)
    snap = tracemalloc.take_snapshot()

    class MyPickler(pickle.Pickler):
        def __init__(self, *args, **kwargs):
            # Swapping the next 2 lines "solves" the memory leak for some reason
            self.dispatch_table = dict()
            super().__init__(*args, **kwargs)

    l=[]
    for i in range(10000):
        if i % 1000 == 0:
            print('='*80)
            snap2 = tracemalloc.take_snapshot()
            stats=snap2.compare_to(snap, 'lineno')
            for s in stats[:10]:
                print(s)
            snap = snap2

        f = io.BytesIO()
        MyPickler(f)
        gc.collect()

The output of the last iteration is as follow. The leak of 562 kiB is apparent:

testmem.py:12: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64 B
/usr/lib/python3.10/tracemalloc.py:125: size=2376 B (-72 B), count=33 (-1), average=72 B
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:502: size=252 B (+28 B), count=9 (+1), average=28 B
/usr/lib/python3.10/tracemalloc.py:498: size=2104 B (+0 B), count=36 (+0), average=58 B
/home/dourai01/Work/lisa/lisa/testmem.py:10: size=1844 B (+0 B), count=9 (+0), average=205 B
/usr/lib/python3.10/tracemalloc.py:193: size=1680 B (+0 B), count=35 (+0), average=48 B
/usr/lib/python3.10/tracemalloc.py:547: size=1256 B (+0 B), count=3 (+0), average=419 B
/usr/lib/python3.10/tracemalloc.py:226: size=832 B (+0 B), count=2 (+0), average=416 B
/usr/lib/python3.10/tracemalloc.py:173: size=800 B (+0 B), count=2 (+0), average=400 B

If "dispatch_table" is set after calling super().__init__, there is no leak anymore:

/usr/lib/python3.10/tracemalloc.py:135: size=740 B (+740 B), count=7 (+7), average=106 B
/usr/lib/python3.10/tracemalloc.py:125: size=2088 B (-656 B), count=29 (-4), average=72 B
/usr/lib/python3.10/tracemalloc.py:136: size=320 B (+320 B), count=1 (+1), average=320 B
/usr/lib/python3.10/tracemalloc.py:132: size=0 B (-256 B), count=0 (-1)
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:498: size=2008 B (+48 B), count=34 (+1), average=59 B
/usr/lib/python3.10/tracemalloc.py:193: size=1584 B (+48 B), count=33 (+1), average=48 B
/usr/lib/python3.10/tracemalloc.py:502: size=196 B (-28 B), count=7 (-1), average=28 B
/usr/lib/python3.10/tracemalloc.py:126: size=84 B (+28 B), count=3 (+1), average=28 B

[tigerinus]

I can verify this also happens on Python 3.9.2

I also have a demonstration of this memory leak at https://github.com/tigerinus/pickle_memory_leak_with_msgspec

[tigerinus]

I can verify this also happens on Python 3.9.2

I also have a demonstration of this memory leak at https://github.com/tigerinus/pickle_memory_leak_with_msgspec

The issue I found turns out to be an issue of msgspec.Struct and should be fixed by jcrist/msgspec#117