gh-115720: Show number of leaks in huntrleaks progress reports #115726

encukou · 2024-02-20T15:14:19Z

Instead of showing a dot for each iteration, show:

'.' for warmup or no leaks
number of leaks for 1-9
'X' if there are more leaks

Also, separate warmup runs by a colon & space.

This allows more insight into what's going on. It's especially useful for rapid iteration (like bisecting).

Issue: Show number of leaks in --huntrleaks #115720

Instead of showing a dot for each iteration, show: - '.' for warmup or no leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: I don't need to wait for the final report to see if the test still leaks.

vstinner · 2024-02-20T15:32:10Z

Lib/test/libregrtest/refleak.py

+                symbol = 'X'
+            if i == warmups:
+                print(' ', end='', file=sys.stderr, flush=True)
+            print(symbol, end='', file=sys.stderr, flush=True)


To avoid any risk of side effect of that code, I suggest to add a get_symbol() function which would look into a pre-computed array for values <= 10. Something like:

symbols = ['.'] + [str(i) for i in range(1, 10)] + ['X'] def get_symbol(value): return symbols[min(max(value, 0), 10)]

vstinner · 2024-02-20T15:33:01Z

Lib/test/libregrtest/refleak.py

+            else:
+                symbol = 'X'
+            if i == warmups:
+                print(' ', end='', file=sys.stderr, flush=True)


This is lovely, thanks :-)

encukou · 2024-02-20T15:44:08Z

Lib/test/libregrtest/refleak.py

+            if total_leaks <= 0:
+                symbol = '.'
+            elif total_leaks < 10:
+                symbol = str(total_leaks)


@vstinner instead of the function, maybe selecting a tuple of single-char strings would work better?

Suggested change

symbol = str(total_leaks)

symbol = (

'.', '1', '2', '3', '4', '5', '6', '7', '8', '9',

)[total_leaks]

vstinner · 2024-02-20T15:44:39Z

The problem is how to display negative numbers: #115725

encukou · 2024-02-20T15:51:06Z

Ah! Good point. Tomorrow I'll update this to make cases like test_uuid looks nicer :)

SIMBA111111 · 2024-02-20T21:53:48Z

veru good

vstinner

I'm not fully convinced that it's a good idea to display intermediate results when we consider that there is no leak. As you saw with test_uuid, leaks can be subtle :-( There are worse cases such as #80741 which are really hard to reproduce and look like false alarms.

We already display values when we consider that there is a leak, and it's maybe enough?

If you have a lot of free time, you can change check_rc_deltas() heuristic to be stricter. But be careful of the rabbit hole, many people lost their mind in the Wonderland of reference leaks :-D

encukou · 2024-02-21T09:07:23Z

What about showing the full result if it's suspicious (i.e. not all zeros), but using the current heuristic for failure (for now)?

12345:6789
X4.6. 16.1
test_uuid leaked [1, 6, -6, 1] references, sum=2 (this is fine)
test_uuid leaked [0, 3, -3, 0] memory blocks, sum=0 (this is fine)

== Tests result: SUCCESS ==

The heuristic is good (for now) for keeping CI green, but for bisecting I want to have all the info.

bedevere-bot · 2024-02-22T09:25:40Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit f2a10a1 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

encukou · 2024-02-26T10:20:37Z

@vstinner Does this sound reasonable?
I'll merge today or tomorrow if there are no objections. We can make it better in future PRs :)

…ythonGH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic.

…ythonGH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic. (cherry picked from commit af5f9d6)

…nch (#117250) * gh-115122: Add --bisect option to regrtest (#115123) * test.bisect_cmd now exit with code 0 on success, and code 1 on failure. Before, it was the opposite. * test.bisect_cmd now runs the test worker process with -X faulthandler. * regrtest RunTests: Add create_python_cmd() and bisect_cmd() methods. (cherry picked from commit 1e5719a) * gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic. (cherry picked from commit af5f9d6) * gh-83434: Disable XML in regrtest when -R option is used (#117232) (cherry picked from commit d52bdfb) --------- Co-authored-by: Petr Viktorin <encukou@gmail.com>

…in branch (pythonGH-117250) * pythongh-115122: Add --bisect option to regrtest (pythonGH-115123) * test.bisect_cmd now exit with code 0 on success, and code 1 on failure. Before, it was the opposite. * test.bisect_cmd now runs the test worker process with -X faulthandler. * regrtest RunTests: Add create_python_cmd() and bisect_cmd() methods. (cherry picked from commit 1e5719a) * pythongh-115720: Show number of leaks in huntrleaks progress reports (pythonGH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic. (cherry picked from commit af5f9d6) * pythongh-83434: Disable XML in regrtest when -R option is used (pythonGH-117232) (cherry picked from commit d52bdfb) --------- (cherry picked from commit 477ef90) Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Petr Viktorin <encukou@gmail.com>

…ain branch (GH-117250) (#117251) [3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch (GH-117250) * gh-115122: Add --bisect option to regrtest (GH-115123) * test.bisect_cmd now exit with code 0 on success, and code 1 on failure. Before, it was the opposite. * test.bisect_cmd now runs the test worker process with -X faulthandler. * regrtest RunTests: Add create_python_cmd() and bisect_cmd() methods. (cherry picked from commit 1e5719a) * gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic. (cherry picked from commit af5f9d6) * gh-83434: Disable XML in regrtest when -R option is used (GH-117232) (cherry picked from commit d52bdfb) --------- (cherry picked from commit 477ef90) Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Petr Viktorin <encukou@gmail.com>

…ythonGH-115726) Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic.

encukou requested a review from vstinner February 20, 2024 15:14

bedevere-app bot added the awaiting core review label Feb 20, 2024

bedevere-app bot mentioned this pull request Feb 20, 2024

Show number of leaks in --huntrleaks #115720

Open

vstinner reviewed Feb 20, 2024

View reviewed changes

Replace str() by tuple lookup

0dd589e

encukou commented Feb 20, 2024

View reviewed changes

vstinner reviewed Feb 21, 2024

View reviewed changes

Show full result if it's not all zeros

f2a10a1

encukou added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Feb 22, 2024

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Feb 22, 2024

encukou merged commit af5f9d6 into python:main Feb 27, 2024
55 checks passed

bedevere-app bot removed the awaiting core review label Feb 27, 2024

encukou deleted the huntrleaks-digits branch February 27, 2024 08:51

encukou mentioned this pull request Feb 27, 2024

Reference count variations after getting __dict__, getattr, and gc.collect #115822

Open

vstinner mentioned this pull request Mar 26, 2024

[3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch #117250

Merged

miss-islington mentioned this pull request Mar 26, 2024

[3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch (GH-117250) #117251

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-115720: Show number of leaks in huntrleaks progress reports #115726

gh-115720: Show number of leaks in huntrleaks progress reports #115726

encukou commented Feb 20, 2024 •

edited by bedevere-app bot

Loading

vstinner Feb 20, 2024

vstinner Feb 20, 2024

encukou Feb 20, 2024

vstinner commented Feb 20, 2024

encukou commented Feb 20, 2024

SIMBA111111 commented Feb 20, 2024

vstinner left a comment •

edited

Loading

encukou commented Feb 21, 2024

bedevere-bot commented Feb 22, 2024

encukou commented Feb 26, 2024

-                symbol = str(total_leaks)
+                symbol = (
+                    '.', '1', '2', '3', '4', '5', '6', '7', '8', '9',
+                    )[total_leaks]

gh-115720: Show number of leaks in huntrleaks progress reports #115726

gh-115720: Show number of leaks in huntrleaks progress reports #115726

Conversation

encukou commented Feb 20, 2024 • edited by bedevere-app bot Loading

vstinner Feb 20, 2024

Choose a reason for hiding this comment

vstinner Feb 20, 2024

Choose a reason for hiding this comment

encukou Feb 20, 2024

Choose a reason for hiding this comment

vstinner commented Feb 20, 2024

encukou commented Feb 20, 2024

SIMBA111111 commented Feb 20, 2024

vstinner left a comment • edited Loading

Choose a reason for hiding this comment

encukou commented Feb 21, 2024

bedevere-bot commented Feb 22, 2024

encukou commented Feb 26, 2024

encukou commented Feb 20, 2024 •

edited by bedevere-app bot

Loading

vstinner left a comment •

edited

Loading