Reference leak regression with Python3.8a3

[kayhayen]

BPO	36725
Nosy	@vstinner, @nanjekyejoannah
PRs	bpo-36725: regrtest: add TestResult type #12960 bpo-36725: Refactor regrtest multiprocessing code #12961 bpo-36719: regrtest -jN no longer stops on crash #13231
Superseder	bpo-36851: Frame stack is not cleaned after execution is finished with return

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-13.15:31:06.436>
created_at = <Date 2019-04-25.20:28:27.757>
labels = ['3.8', 'performance']
title = 'Reference leak regression with Python3.8a3'
updated_at = <Date 2019-05-13.17:17:56.569>
user = 'https://bugs.python.org/kayhayen'

bugs.python.org fields:

activity = <Date 2019-05-13.17:17:56.569>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-13.15:31:06.436>
closer = 'vstinner'
components = []
creation = <Date 2019-04-25.20:28:27.757>
creator = 'kayhayen'
dependencies = []
files = []
hgrepos = []
issue_num = 36725
keywords = ['patch']
message_count = 10.0
messages = ['340861', '340876', '340883', '340888', '340890', '340901', '340917', '340920', '340921', '342340']
nosy_count = 3.0
nosy_names = ['vstinner', 'kayhayen', 'nanjekyejoannah']
pr_nums = ['12960', '12961', '13231']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '36851'
type = 'resource usage'
url = 'https://bugs.python.org/issue36725'
versions = ['Python 3.8']

[kayhayen]

Much like bpo-9366 the same file can be used. This reference leaks according to Nuitka comparative testing:

simpleFunction59: FAILED 129511 129512 leaked 1

def simpleFunction59():
    a = 3
    b = 5

try:
    a = a * 2

        return a
    finally:
        return a / b

I would be guessing, that you are leaking the return value when returning again.

[vstinner]

New changeset 4d29983 by Victor Stinner in branch 'master':
bpo-36725: regrtest: add TestResult type (GH-12960)
4d29983

[vstinner]

New changeset 3cde440 by Victor Stinner in branch 'master':
bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
3cde440

[vstinner]

Sorry, the two commits are related to bpo-36719 not to this issue.

[vstinner]

I wrote an unit test to be able to re-use regrtest memory leak hunter:

import unittest

def simpleFunction59():
    a = 1
    try:
        return a
    finally:
        return a

class Tests(unittest.TestCase):
    def test_bug(self):
        for _ in range(10):
            simpleFunction59()

I confirm that there is a leak:

$ ./python -m test -R 3:3 test_bug
...
test_bug leaked [10, 10, 10] references, sum=30
...

[vstinner]

New changeset 1069d38 by Victor Stinner in branch '3.7':
[3.7] bpo-36719: sync regrtest with master branch (GH-12967)
1069d38

[nanjekyejoannah]

Let me work on this.

[nanjekyejoannah]

@ victor, I have tried your example but I can not reproduce the leak as you did above. May be am missing something?

I used some other tool and got this summary when I was investigating this:

                        types |   # objects |   total size

================================= | =========== | ============
<class 'str | 3916 | 361.40 KB
<class 'list | 2513 | 237.41 KB
<class 'dict | 197 | 120.28 KB
<class 'code | 617 | 86.79 KB
<class 'type | 81 | 80.71 KB
<class 'int | 497 | 13.60 KB
<class 'tuple | 199 | 12.80 KB
function (init) | 61 | 8.10 KB
<class 'weakref | 87 | 6.80 KB
<class 'getset_descriptor | 67 | 4.71 KB
<class 'collections.OrderedDict | 3 | 4.22 KB
<class 'enum.EnumMeta | 3 | 2.98 KB
function (call) | 14 | 1.86 KB
<class 'cell | 37 | 1.73 KB
<enum 'Signals | 32 | 1.62 KB

Am still investigating though.

[nanjekyejoannah]

I am able to reproduce it finally.

Run tests sequentially
0:00:00 load avg: 0.21 [1/1] test_bug
beginning 6 repetitions
123456
......
test_bug leaked [10, 10, 10] references, sum=30
test_bug failed

== Tests result: FAILURE ==

1 test failed:
test_bug

Total duration: 220 ms
Tests result: FAILURE

[vstinner]

This leak has been fixed by:

commit f00828a (refs/bisect/bad)
Author: Pablo Galindo <Pablogsal@gmail.com>
Date: Thu May 9 16:52:02 2019 +0100

bpo-36851: Clean the frame stack if the execution ends with a return and the stack is not empty (GH-13191)