Remove COUNT_ALLOCS special build #83670
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2020-02-03.14:21:24.624> created_at = <Date 2020-01-29.18:16:16.453> labels = ['build', '3.9'] title = 'Remove COUNT_ALLOCS special build' updated_at = <Date 2020-02-03.14:21:24.623> user = 'https://github.com/vstinner'
activity = <Date 2020-02-03.14:21:24.623> actor = 'vstinner' assignee = 'none' closed = True closed_date = <Date 2020-02-03.14:21:24.624> closer = 'vstinner' components = ['Build'] creation = <Date 2020-01-29.18:16:16.453> creator = 'vstinner' dependencies =  files =  hgrepos =  issue_num = 39489 keywords = ['patch'] message_count = 12.0 messages = ['360978', '360979', '360981', '360985', '360986', '360987', '360988', '361036', '361046', '361051', '361294', '361297'] nosy_count = 3.0 nosy_names = ['rhettinger', 'vstinner', 'serhiy.storchaka'] pr_nums = ['18259'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue39489' versions = ['Python 3.9']
The text was updated successfully, but these errors were encountered:
Python has a COUNT_ALLOCS special build which adds sys.getcounts() function and shows statistics on Python types at exit if -X showalloccount command line option is used.
I never ever used this feature and I don't know anyone using it.
But "#ifdef COUNT_ALLOCS" code is scattered all around the code. It requires maintenance. I propose to remove the code to ease maintenance.
Attached PR shows how much code is requires to support this special build.
There are now more advanced tools to have similar features:
The previous large change related to COUNT_ALLOCS was done in Python 3.6 by bpo-23034:
"The output of a special Python build with defined COUNT_ALLOCS, SHOW_ALLOC_COUNT or SHOW_TRACK_COUNT macros is now off by default. It can be re-enabled using the -X showalloccount option. It now outputs to stderr instead of stdout. (Contributed by Serhiy Storchaka in bpo-23034.)"
See also bpo-19527 which discussed COUNT_ALLOCS in 2013.
COUNT_ALLOCS build is documented in Fedora "DebugPythonStacks" wiki page:
As far as I known, the Fedora package of Python is the most known user of COUNT_ALLOCS special build, and maybe the only user.
The Fedora package of Python 2.7 builds two binaries:
Sadly, in practice, python2.7-debug is basically unusable since C extensions providing by Fedora are not compatible: the release and the debug modes have a different ABI. I only fixed this recently in Python 3.8.
It uses COUNT_ALLOCS since the following patch (package version 2.6.5-10, in 2010) written by Dave Malcolm.
(1) There is a downstream patch adding --with-count-allocs flag to configure. Patch added to package version 2.6.5-10, in 2010:
(2) Another downstream patch adds PYTHONDUMPCOUNTS environment variable to only dump statistics if the variable is set. Patch added to package version 2.7-8, in 2010:
=> this patch was proposed upstream in bpo-19527. I merged it in Python 2.7.15 (in 2017), but with a different environment variable name: PYTHONSHOWALLOCCOUNT.
(3) Finally, a 3rd downstream patch fix test_abc when COUNT_ALLOCS is defined. Patch added to package version 2.7.1-1, in 2010:
=> Bohuslav "Slavek" Kabrda created bpo-19527 to propose to make multiple test fixes upstream. Serhiy Storchaka modified the proposed patch by adding @test.support.requires_type_collecting decorator. Change merged in 3.5 and default branches, but not in 2.7.
Another example of Fedora contribution: bpo-31692 reports that test_regrtest fails when Python 2.7.14 is built with COUNT_ALLOCS.
Now, the interesting part.
IMHO COUNT_ALLOCS was basically kept between 2010 and 2015 because "it was there" and nobody asked if it still made sense to use it. It wasn't too expensive to maintain in the Fedora package... until someone asked if it's still worth it to maintain it in 2015.
The COUNT_ALLOCS macro has been removed from the Fedora package of Python 3 in 2015:
Extract of the bugzilla, "Reply from Dave Malcolm":
I don't think this patch ever really bought us much, and it sounds like
It confirms what I wrote in the initial message of this issue: there are now better tool to track Python memory leaks.
Supporting COUNT_ALLOC in the Fedora package was motivated by Dave Malcolm use case of tracking memory leak, but even Dave wrote: "I don't think this patch ever really bought us much".
Python 2.7.15 got a PYTHONSHOWALLOCCOUNT environment variable to dump statistics on types at exit, if Python is built with COUNT_ALLOCS macro defined.
Example with Fedora python2.7-debug (package python2-debug-2.7.17-1.fc31.x86_64):
$ PYTHONSHOWALLOCCOUNT=1 python2.7-debug -c pass exceptions.ImportError alloc'd: 2, freed: 2, max in use: 1 symtable entry alloc'd: 3, freed: 3, max in use: 1 enumerate alloc'd: 2, freed: 2, max in use: 1 dict alloc'd: 459, freed: 220, max in use: 346 str alloc'd: 13981, freed: 12142, max in use: 5991 tuple alloc'd: 5088, freed: 4096, max in use: 2337 (...) fast tuple allocs: 2434, empty: 1753 fast int allocs: pos: 875, neg: 74 null strings: 81, 1-strings: 328
I looked for "Python COUNT_ALLOCS" on the Internet. This special build seems to be very badly documented on the Internet. Outside Python own documentation, I found almost zero reference to it. IMHO it's basically unused.
(*) bpo-33058: Two years ago (2018), attempt to make COUNT_ALLOCS build ABI compatible with release build. Extract:
(*) Documentation of all Python builds:
The COUNT_ALLOCS feature itself is quite old. It was added 27 years ago (in 1993):
I used COUNT_ALLOCS for curiosity. But I needed slightly different information, so in any case I patched the code.
AFAIK COUNT_ALLOCS is used in some large projects (maybe Fedora). It was already discussed somewhere on the tracker, but I have no a link.
I started a thread on the python-dev mailing list:
Did you use it recently? I understand that the unmodified feature didn't help for your use case. Was it to debug memory leaks?
Did you read previous comments, especially msg360985? It's no longer used in Fedora :-) Technically, it's still used in the python2 package, but it's gone from python3 package. But this issue is about Python 3.9, not Python 2 ;-)
FYI I'm now working in the Python Maintenance team at Red Hat which maintains python2 and python3 packages of Fedora.
By the way, Charalampos Stratakis is also in my team and he replied on python-dev:
"I've never used this feature and it was quite the hassle to maintain those patches downstream, so in my biased opinion, I would welcome the removal."
It was many years ago, and I used it to get some statistics about using Python objects of some types. But it did not work as I wanted for cached integers etc. In any case I can patch Python again if I need such information.
Seems Fedora was the main user of this feature. If it is no longer used in Fedora, I think it can be safely removed.