Skip to content

Conversation

lkollar
Copy link
Contributor

@lkollar lkollar commented Sep 13, 2025

The stack collector base class keeps all frames until export() is called, which causes significant unnecessary memory usage. Instead, we can process the frames on the fly in the collect call by dispatching the aggregation logic to the subclass through the process_frames method.

Copilot

This comment was marked as off-topic.

@python python deleted a comment from Copilot AI Sep 13, 2025
@python python deleted a comment from Copilot AI Sep 13, 2025
@pablogsal pablogsal force-pushed the profiling/reduce-memory-usage branch 2 times, most recently from eb1bffb to a89c852 Compare September 14, 2025 02:51
@pablogsal
Copy link
Member

pablogsal commented Sep 14, 2025

Amazing! I have pushed a commit adding an initial version of the string table for the flame graph. We can improve that in subsequent PRs

@pablogsal pablogsal enabled auto-merge (squash) September 14, 2025 20:52
@pablogsal pablogsal disabled auto-merge September 14, 2025 20:52
@pablogsal
Copy link
Member

Will fix merge conflicts tomorrow

lkollar and others added 2 commits September 14, 2025 23:17
The stack collector base class keeps all frames until export() is
called, which causes significant unnecessary memory usage. Instead, we
can process the frames on the fly in the collect call by dispatching the
aggregation logic to the subclass through the process_frames method.
@pablogsal pablogsal force-pushed the profiling/reduce-memory-usage branch from a89c852 to 52c9867 Compare September 14, 2025 22:18
@pablogsal pablogsal merged commit 3e06cfc into python:main Sep 14, 2025
43 checks passed
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit 3e06cfc.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2071) and take a look at the build logs.
  4. Check if the failure is related to this commit (3e06cfc) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2071

Failed tests:

  • test_free_threading

Test leaking resources:

  • test_free_threading: file descriptors

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 22, done.        
remote: Counting objects:   4% (1/22)        
remote: Counting objects:   9% (2/22)        
remote: Counting objects:  13% (3/22)        
remote: Counting objects:  18% (4/22)        
remote: Counting objects:  22% (5/22)        
remote: Counting objects:  27% (6/22)        
remote: Counting objects:  31% (7/22)        
remote: Counting objects:  36% (8/22)        
remote: Counting objects:  40% (9/22)        
remote: Counting objects:  45% (10/22)        
remote: Counting objects:  50% (11/22)        
remote: Counting objects:  54% (12/22)        
remote: Counting objects:  59% (13/22)        
remote: Counting objects:  63% (14/22)        
remote: Counting objects:  68% (15/22)        
remote: Counting objects:  72% (16/22)        
remote: Counting objects:  77% (17/22)        
remote: Counting objects:  81% (18/22)        
remote: Counting objects:  86% (19/22)        
remote: Counting objects:  90% (20/22)        
remote: Counting objects:  95% (21/22)        
remote: Counting objects: 100% (22/22)        
remote: Counting objects: 100% (22/22), done.        
remote: Compressing objects:  12% (1/8)        
remote: Compressing objects:  25% (2/8)        
remote: Compressing objects:  37% (3/8)        
remote: Compressing objects:  50% (4/8)        
remote: Compressing objects:  62% (5/8)        
remote: Compressing objects:  75% (6/8)        
remote: Compressing objects:  87% (7/8)        
remote: Compressing objects: 100% (8/8)        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 12 (delta 9), reused 5 (delta 4), pack-reused 0 (from 0)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to '3e06cfcaeee31c2a6e9befe5ea54d9beaa2c0434'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3e06cfcaeee gh-135953: Reduce memory usage of stack collectors (#138875)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

make: *** [Makefile:2486: buildbottest] Error 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants