Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Show number of leaks in --huntrleaks #115720

Open
encukou opened this issue Feb 20, 2024 · 1 comment
Open

Show number of leaks in --huntrleaks #115720

encukou opened this issue Feb 20, 2024 · 1 comment
Assignees
@encukou
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@encukou
Copy link
Member

encukou commented Feb 20, 2024 •
edited by bedevere-app bot
Loading

Feature or enhancement

Currently, -R (--huntrleaks) only displays the reference counts when the test fails. It also uses a generous heuristic, which avoid false positives, but might hide issues.

To get more insight into what's going on, I propose to show the number of leaks instead of the dot, with

  • . instead of 0, so leaks stand out
  • X instead of 10 or more, to keep the display as a single digit

Additionally, I'd like to separate warmup runs, so you know when to start worrying.

The output would look like this:

beginning 9 repetitions. Showing number of leaks (. for zero, X for 10 or more)
12345:6789
XX.1. ....

meaning the first 2 warmups “leaked” a lot, then one more had a single “leak”, and there were no more leaks.

Linked PRs
@encukou encukou added type-feature A feature request or enhancement tests Tests in the Lib/test dir labels Feb 20, 2024
@encukou encukou self-assigned this Feb 20, 2024
@encukou encukou mentioned this issue Feb 20, 2024
Closed
@bedevere-app bedevere-app bot mentioned this issue Feb 20, 2024
Merged
encukou added a commit that referenced this issue Feb 27, 2024
@encukou
gh-115720: Show number of leaks in huntrleaks progress reports (GH-11…
af5f9d6 
…5726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.
@encukou
Copy link
Member Author

encukou commented Feb 27, 2024

Merged.
The behaviour is slightly different, to account for negative “leak counts”; for details see af5f9d6. (Or a test run -- the output should be self-descriptive.)

I plan to leave this in main for a while before deciding about backports.

woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@encukou @woodruffw
pythongh-115720: Show number of leaks in huntrleaks progress reports (p…
f668718 
…ythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@encukou @adorilson
pythongh-115720: Show number of leaks in huntrleaks progress reports (p…
9063b8f 
…ythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.
vstinner pushed a commit to vstinner/cpython that referenced this issue Mar 26, 2024
@encukou @vstinner
pythongh-115720: Show number of leaks in huntrleaks progress reports (p…
2633b09 
…ythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)
vstinner pushed a commit to vstinner/cpython that referenced this issue Mar 26, 2024
@encukou @vstinner
pythongh-115720: Show number of leaks in huntrleaks progress reports (p…
01436c5 
…ythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)
@vstinner vstinner mentioned this issue Mar 26, 2024
Merged
vstinner added a commit that referenced this issue Mar 26, 2024
@vstinner @encukou
[3.12] gh-83434: Sync libregrtest and test_regrtest with the main bra…
477ef90 
…nch (#117250)

* gh-115122: Add --bisect option to regrtest (#115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* gh-83434: Disable XML in regrtest when -R option is used (#117232)

(cherry picked from commit d52bdfb)

---------

Co-authored-by: Petr Viktorin <encukou@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 26, 2024
@vstinner @encukou @miss-islington
[3.12] pythongh-83434: Sync libregrtest and test_regrtest with the ma…
23a5deb 
…in branch (pythonGH-117250)

* pythongh-115122: Add --bisect option to regrtest (pythonGH-115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* pythongh-115720: Show number of leaks in huntrleaks progress reports (pythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* pythongh-83434: Disable XML in regrtest when -R option is used (pythonGH-117232)

(cherry picked from commit d52bdfb)

---------

(cherry picked from commit 477ef90)

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
@miss-islington miss-islington mentioned this issue Mar 26, 2024
Merged
vstinner added a commit that referenced this issue Mar 26, 2024
@miss-islington @vstinner @encukou
[3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the m…
04b26ee 
…ain branch (GH-117250) (#117251)

[3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch (GH-117250)

* gh-115122: Add --bisect option to regrtest (GH-115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* gh-83434: Disable XML in regrtest when -R option is used (GH-117232)

(cherry picked from commit d52bdfb)

---------

(cherry picked from commit 477ef90)

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@encukou @diegorusso
pythongh-115720: Show number of leaks in huntrleaks progress reports (p…
4221533 
…ythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@encukou encukou

Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@encukou