Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-109786: Fix leaks when re-enter itertools.pairwise.__next__() #109788

Merged
merged 6 commits into from Dec 4, 2023
Merged

gh-109786: Fix leaks when re-enter itertools.pairwise.__next__() #109788

merged 6 commits into from Dec 4, 2023

Conversation

serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Sep 23, 2023 •
edited by bedevere-app bot

@serhiy-storchaka serhiy-storchaka added needs backport to 3.11 only security fixes needs backport to 3.12 bug and security fixes labels Sep 23, 2023
@bedevere-app bedevere-app bot mentioned this pull request Sep 23, 2023
Closed
serhiy-storchaka
serhiy-storchaka commented Sep 23, 2023
View reviewed changes
Lib/test/test_itertools.py Outdated Show resolved Hide resolved
Modules/itertoolsmodule.c Outdated
return NULL;
}
}
else {
Py_INCREF(old);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other simple solution is setting po->old = NULL instead of increfing old, so old holds the only reference. It only differs from the present solution in case check(3, ...), and I tested much more test cases than present in this PR, including multiple re-entering points.

@rhettinger
Copy link
Contributor

How does the "borrowed reference" arise? My understanding that an iternext call returns a new reference (at least that is what PyIter_Next() assumes.

@serhiy-storchaka
Copy link
Member Author

po->old owns a reference. old holds a borrowed reference on it when Py_TYPE(it)->tp_iternext is called.

@serhiy-storchaka
Copy link
Member Author

Now the current result is preserved in all test cases.

I also found and fixed a crash due to it holding a borrowed reference. I am not sure, but the tp_iternext calls may need Py_INCREF/Py_DECREF around it.

@serhiy-storchaka
Copy link
Member Author

@rhettinger, I am going to merge this if you have no questions or objections.

There are many ways to fix leaks, and I tested many variants. The proposed one is the only variant that produces the same result in all tested weird cases (but without leaking of course).

@serhiy-storchaka serhiy-storchaka enabled auto-merge (squash) December 4, 2023 11:23
@serhiy-storchaka serhiy-storchaka merged commit 6ca9d3e into python:main Dec 4, 2023
32 checks passed
@miss-islington-app
Copy link

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

@bedevere-app
Copy link

bedevere-app bot commented Dec 4, 2023

GH-112699 is a backport of this pull request to the 3.12 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Dec 4, 2023
@serhiy-storchaka @miss-islington
pythongh-109786: Fix leaks and crash when re-enter itertools.pairwise…
b470925 
….__next__() (pythonGH-109788)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app bedevere-app bot removed the needs backport to 3.12 bug and security fixes label Dec 4, 2023
@bedevere-app
Copy link

bedevere-app bot commented Dec 4, 2023

GH-112700 is a backport of this pull request to the 3.11 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Dec 4, 2023
@serhiy-storchaka @miss-islington
pythongh-109786: Fix leaks and crash when re-enter itertools.pairwise…
838a9d1 
….__next__() (pythonGH-109788)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app bedevere-app bot removed the needs backport to 3.11 only security fixes label Dec 4, 2023
serhiy-storchaka added a commit that referenced this pull request Dec 4, 2023
@miss-islington @serhiy-storchaka
[3.12] gh-109786: Fix leaks and crash when re-enter itertools.pairwis…
55896f4 
…e.__next__() (GH-109788) (GH-112699)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this pull request Dec 4, 2023
@miss-islington @serhiy-storchaka
[3.11] gh-109786: Fix leaks and crash when re-enter itertools.pairwis…
e3d380d 
…e.__next__() (GH-109788) (GH-112700)

(cherry picked from commit 6ca9d3e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@serhiy-storchaka @aisk
pythongh-109786: Fix leaks and crash when re-enter itertools.pairwise…
b92fee0 
….__next__() (pythonGH-109788)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rhettinger rhettinger Awaiting requested review from rhettinger rhettinger is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@serhiy-storchaka @rhettinger