Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-entering pairwise.__next__() leaks references #109786

Closed
serhiy-storchaka opened this issue Sep 23, 2023 · 3 comments
Closed

Re-entering pairwise.__next__() leaks references #109786

serhiy-storchaka opened this issue Sep 23, 2023 · 3 comments
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 new features, bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Sep 23, 2023

Re-entering the __next__() method of itertools.pairwise leaks references, because old hold a borrowed reference when the __next__() method of the underlying iterator is called. It may potentially lead to the use of a freed memory and a crash, but I only have reproducer for leaks.

Even if the re-entrant call is not very meaningful, it should correctly count references. There are several ways to fix this issue. I choose the simplest one which matches the current results (only without leaks) in most of simple tests, even if there is no other reasons to prefer these results.

cc @rhettinger

Linked PRs

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Sep 23, 2023
@serhiy-storchaka serhiy-storchaka added type-bug An unexpected behavior, bug, or error 3.11 only security fixes 3.12 bugs and security fixes 3.13 new features, bugs and security fixes labels Sep 23, 2023
@rhettinger
Copy link
Contributor

Let's get the other issue fixed first. This one is somewhat exotic and the PR makes a mess of the code.

@pochmann
Copy link
Contributor

pochmann commented Sep 23, 2023

I don't see your leak reproducer, wrote one based on your test:

Leak reproducer

Whole code at Attempt This Online!

incref(x) artificially increases the refcount of x:

def incref(x):
    class I:
        count = 0
        def __iter__(self):
            return self
        def __next__(self):
            self.count += 1
            if self.count == 1:
                return next(pairs)
            if self.count == 3:
                return x
            return None
    pairs = pairwise(I())
    next(pairs)

Demo increasing an object's refcount by 10000:

x = object()
print(f'refcount before:', sys.getrefcount(x))
for _ in range(10000):
    incref(x)
gc.collect()
print('refcount after:', sys.getrefcount(x))

Output:

refcount before: 2
refcount after: 10002

Demo leaking 10000 objects of 10 kB each:

tracemalloc.start()
for _ in range(10000):
    incref(bytes(10000))
gc.collect()
print('leaked memory:', tracemalloc.get_traced_memory()[0] // 10**6, 'MB')

Output:

leaked memory: 100 MB

@serhiy-storchaka
Copy link
Member Author

I added them as tests. You can run tests with the -R option to see leaks.

@iritkatriel iritkatriel added the stdlib Python modules in the Lib dir label Nov 25, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Dec 4, 2023
….__next__() (pythonGH-109788)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Dec 4, 2023
….__next__() (pythonGH-109788)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Dec 4, 2023
…e.__next__() (GH-109788) (GH-112699)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Dec 4, 2023
…e.__next__() (GH-109788) (GH-112700)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 new features, bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

5 participants
@pochmann @iritkatriel @rhettinger @serhiy-storchaka and others