Enhance Python's Memory Instrumentation with COUNT_ALLOCS

[eduardo-elizondo]

BPO	33058
Nosy	@vstinner, @serhiy-storchaka, @eduardo-elizondo
PRs	bpo-33058: Enhanced COUNT_ALLOCS to be ABI Compatible #6091

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-03-12.19:31:15.946>
labels = ['type-feature', '3.8']
title = "Enhance Python's Memory Instrumentation with COUNT_ALLOCS"
updated_at = <Date 2018-03-12.22:05:43.281>
user = 'https://github.com/eduardo-elizondo'

bugs.python.org fields:

activity = <Date 2018-03-12.22:05:43.281>
actor = 'eelizondo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2018-03-12.19:31:15.946>
creator = 'eelizondo'
dependencies = []
files = []
hgrepos = []
issue_num = 33058
keywords = ['patch']
message_count = 4.0
messages = ['313684', '313685', '313686', '313691']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'eelizondo']
pr_nums = ['6091']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33058'
versions = ['Python 3.8']

[eduardo-elizondo]

[WIP]

[eduardo-elizondo]

Currently, Python has very few instrumentation when it comes to the types of allocations that it's doing. For instance, we currently can't identify how many times an object uses free lists vs actual mallocs.

Currently, there's a special build which can be used by compiling with "-DCOUNT_ALLOCS". However, this build is not ABI compatible with extensions. Meaning you have to recompile all the modules that are used. Doing this on a large scale scenario (1000+ modules) is not feasible.

Thus, I propose the following enhancements:

• Make COUNT_ALLOCS ABI compatible

Then:

• Expand the counters to not only track allocs and frees but also distinguish between:
• GC Counts: Allocs/Frees coming from PyObject_GC_Malloc PyObject_GC_Del
• Memory Counts: Allocs/Frees coming from PyMem_Malloc/PyObject_Malloc PyObject_Free (modulo GC objects).
• Free Lists: "Allocs/Frees" coming from specific object's free_lists/caches.
• Misc: Extra kinds of "Allocs/Frees" such as code's zombie frame.

[serhiy-storchaka]

Could tracemalloc help you?

[eduardo-elizondo]

@serhiy.storchaka tracemalloc can't distinguish between the usage of gc allocs, normal mallocs, and free list reuse.

[iritkatriel]

The PR was closed by the OP in 2019. Closing this as well.

[vstinner]

The COUNT_ALLOCS special build was removed in Python 3.9: #83670