Failure to build with address sanitizer

[btharper]

BPO	36356
Nosy	@vstinner, @ned-deily, @matrixise, @miss-islington, @btharper
PRs	bpo-36356: Fix leak of locale string #12425 bpo-36356: Release Unicode interned strings on Valgrind #12431 bpo-36333, bpo-36356: Fix _PyEval_FiniThreads() #12432 bpo-36356: pymain_free() calls _PyRuntime_Finalize() #12435 [3.7] bpo-36356: pymain_free() calls _PyRuntime_Finalize() #12436 bpo-36356: Fix leak when updating search path in config #12444 bpo-36356: Destroy the GIL at exit #12453 bpo-36356: Fix _PyCoreConfig_Read() #12454 bpo-36356: pymain_exit_error() only call pymain_free() for exit #12968 Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" #13006 bpo-36356: Fix memory leak in _asynciomodule.c during "setup.py build_ext" #16598 [3.8] bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598) #16622

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-08.13:03:16.055>
created_at = <Date 2019-03-19.03:47:57.667>
labels = ['build', '3.8']
title = 'Failure to build with address sanitizer'
updated_at = <Date 2019-10-08.13:03:16.054>
user = 'https://github.com/btharper'

bugs.python.org fields:

activity = <Date 2019-10-08.13:03:16.054>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-08.13:03:16.055>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-03-19.03:47:57.667>
creator = 'btharper'
dependencies = []
files = []
hgrepos = []
issue_num = 36356
keywords = ['patch']
message_count = 28.0
messages = ['338315', '338326', '338329', '338338', '338341', '338352', '338353', '338357', '338358', '338364', '338378', '338431', '338774', '338776', '338907', '340907', '340912', '341060', '341061', '341064', '341065', '341066', '354107', '354116', '354119', '354123', '354200', '354201']
nosy_count = 5.0
nosy_names = ['vstinner', 'ned.deily', 'matrixise', 'miss-islington', 'btharper']
pr_nums = ['12425', '12431', '12432', '12435', '12436', '12444', '12453', '12454', '12968', '13006', '16598', '16622']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue36356'
versions = ['Python 3.8']

[btharper]

Trying to run make after './configure --with-address-sanitizer --with-pydebug' fails with leak of locale string

[matrixise]

just for me, btharper, which system do you use? because I have a fedora 29 and when I try to compile with these flags, getaddrinfo is not found.

[matrixise]

ok, found

I have to disable ipv6 and install libasan

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi

=================================================================
==5408==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 34 byte(s) in 1 object(s) allocated from:
#0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
#1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
#2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
#3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
#4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
#5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
#6 0x652f5f in get_ctype_locale Python/preconfig.c:483
#7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
#8 0x616cde in _PyCoreConfig_ReadPreConfig Python/coreconfig.c:1342
#9 0x620564 in _PyCoreConfig_Read Python/coreconfig.c:1378
#10 0x65ef06 in pyinit_coreconfig Python/pylifecycle.c:740
#11 0x65fa05 in _Py_InitializeCore Python/pylifecycle.c:786
#12 0x42abba in pymain_init Modules/main.c:370
#13 0x42addc in pymain_main Modules/main.c:889
#14 0x42b2bb in _Py_UnixMain Modules/main.c:940
#15 0x4258ee in main Programs/python.c:16
#16 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 34 byte(s) in 1 object(s) allocated from:
#0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
#1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
#2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
#3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
#4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
#5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
#6 0x652f5f in get_ctype_locale Python/preconfig.c:483
#7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
#8 0x6585f8 in pyinit_preconfig Python/pylifecycle.c:723
#9 0x65f8b0 in _Py_InitializeCore Python/pylifecycle.c:781
#10 0x42abba in pymain_init Modules/main.c:370
#11 0x42addc in pymain_main Modules/main.c:889
#12 0x42b2bb in _Py_UnixMain Modules/main.c:940
#13 0x4258ee in main Programs/python.c:16
#14 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: 68 byte(s) leaked in 2 allocation(s).
generate-posix-vars failed
make: *** [Makefile:586: pybuilddir.txt] Error 1

[matrixise]

your PR seems to be fine.

but I continue to get two refleaks with valgrind but don't worry, @vstinner is working on these refleaks.

==5440== 64 bytes in 1 blocks are possibly lost in loss record 1 of 2
==5440== at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440== by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440== by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440== by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440== by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440== by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440== by 0x5154F0: _PyRuntimeState_Init_impl (pystate.c:52)
==5440== by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440== by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440== by 0x4236FE: pymain_init (main.c:339)
==5440== by 0x423980: pymain_main (main.c:889)
==5440== by 0x423A6A: _Py_UnixMain (main.c:940)
==5440==
==5440== 64 bytes in 1 blocks are possibly lost in loss record 2 of 2
==5440== at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440== by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440== by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440== by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440== by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440== by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440== by 0x515506: _PyRuntimeState_Init_impl (pystate.c:58)
==5440== by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440== by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440== by 0x4236FE: pymain_init (main.c:339)
==5440== by 0x423980: pymain_main (main.c:889)
==5440== by 0x423A6A: _Py_UnixMain (main.c:940)

[vstinner]

New changeset e130a07 by Victor Stinner (btharper) in branch 'master':
bpo-36356: Fix memory leak in _PyPreConfig_Read() (GH-12425)
e130a07

[vstinner]

New changeset a712679 by Victor Stinner in branch 'master':
bpo-36333, bpo-36356: Fix _PyEval_FiniThreads() (GH-12432)
a712679

[vstinner]

New changeset fecc4f2 by Victor Stinner in branch 'master':
bpo-36356: Release Unicode interned strings on Valgrind (bpo-12431)
fecc4f2

[vstinner]

New changeset f5f336a by Victor Stinner in branch 'master':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12435)
f5f336a

[vstinner]

New changeset 935250d by Victor Stinner in branch '3.7':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12436)
935250d

[vstinner]

At least, "./python -V" no longer leaks at commit dcf6171.

$ ./configure --with-valgrind
$ make
$ valgrind ./python -V
==9553== Memcheck, a memory error detector
==9553== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==9553== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==9553== Command: ./python -V
==9553== 
Python 3.8.0a2+
==9553== 
==9553== HEAP SUMMARY:
==9553==     in use at exit: 0 bytes in 0 blocks
==9553==   total heap usage: 33 allocs, 33 frees, 5,933 bytes allocated
==9553== 
==9553== All heap blocks were freed -- no leaks are possible
==9553== 
==9553== For counts of detected and suppressed errors, rerun with: -v
==9553== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[btharper]

I'm on Ubuntu 18.10/amd64 compiling with Ubuntu's GCC 8.2.0, I know there's some libraries that are missing dependencies (including bz2 and sqlite) so I may have missed the ipv6 dependencies as well.

My eventual goal was to be able to build a pgo optimized build of cpython with the address sanitizer turned on. Currently on my build of f5f336a I see 12 failing tests and a heap use after free as part of the ctypes test suite.

The ctypes use after free has already been submitted as bpo-36253 since it was the only thing asan caught during tests with the leak sanitizer turned off.

[vstinner]

New changeset 4a1468e by Victor Stinner in branch 'master':
bpo-36356: Fix _PyCoreConfig_Read() (GH-12454)
4a1468e

[ned-deily]

For what it's worth, with current HEAD of master (commit 62be338 which includes previously merged PRs from this issue), current macOS clang with address sanitizer and pydebug enabled gets an assertion failure in parsetok.c. Current HEAD of 3.7 does not.

$ ./configure --with-address-sanitizer --prefix=/tmp/d --with-pydebu
$ make -j3
[...]
./python -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup -fsanitize=address    ' OPT='-g -O0 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python -E ./setup.py  build
Assertion failed: ((intptr_t)(int)(a - line_start) == (a - line_start)), function parsetok, file Parser/parsetok.c, line 308.
/bin/sh: line 1: 95059 Abort trap: 6           CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup -fsanitize=address    ' OPT='-g -O0 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py $quiet build
make: *** [sharedmods] Error 134
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
# same result with explicit CC=clang

[btharper]

I'm not sure about the change from 3.7 to master, but that assertion may be more likely to happen while instrumented with ASAN due to the extra space reserved between heap objects. As far as I can tell it's just expecting that the offset of two pointers will fit within an int instead of a intptr_t (4 bytes versus 8 bytes on my system).

For me running the test_pydoc from the test suite fails reliably with the parsetok.c assertion failure, but can be made to pass with a smaller ASAN redzone. The redzone must be a power of 2 and at least 16, default of 2048.

Fails:
ASAN_OPTIONS="max_redzone=256" ./python Tools/scripts/run_tests.py test_pydoc

Passes:
ASAN_OPTIONS="max_redzone=128" ./python Tools/scripts/run_tests.py test_pydoc

Values of 16, 32, and 64 also pass.

[matrixise]

@ned, I have found a solution on Linux, will try on this evening with my
macOS laptop.
I hope to come with a solution on this evening or tomorrow, but need to
check my proto.

[vstinner]

New changeset 4cb525a by Victor Stinner in branch 'master':
bpo-36356: pymain_exit_error() only call pymain_free() for exit (GH-12968)
4cb525a

[vstinner]

See also bpo-36724 "Clear _PyRuntime at exit".

[vstinner]

New changeset b36e5d6 by Victor Stinner in branch 'master':
bpo-36356: Destroy the GIL at exit (GH-12453)
b36e5d6

[vstinner]

The initial issue has been fixed, so I close the issue. I prefer to get more specific issues like bpo-36724 "Clear _PyRuntime at exit" to following changes.

Thanks Ben Harper for the bug report and the fix.

Note: Ben, I like your songs! Good to know that you chose to contribute to Python in your free time! ;-)

[vstinner]

Oh. test_daemon_threads_shutdown_stdout_deadlock of test_io fails randomly on buildbots.

I ran "./python -m test test_io -m '*daemon*' -F -j4" for 5 minutes on my Fedora 29 before merging my PR, but I didn't get any failure :-(

======================================================================
FAIL: test_daemon_threads_shutdown_stdout_deadlock (test.test_io.CMiscIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_io.py", line 4186, in test_daemon_threads_shutdown_stdout_deadlock
    self.check_daemon_threads_shutdown_deadlock('stdout')
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_io.py", line 4177, in check_daemon_threads_shutdown_deadlock
    self.assertIn("Fatal Python error: could not acquire lock "
AssertionError: "Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stdout>'> at interpreter shutdown, possibly due to daemon threads" not found in 'Fatal Python error: PyCOND_FINI(_PyRuntime.ceval.gil.cond) failed'

Failures:

AMD64 Fedora Rawhide Clang Installed 3.x: fail then pass
https://buildbot.python.org/all/#/builders/188/builds/93

s390x Debian 3.x: fail then pass
https://buildbot.python.org/all/#/builders/13/builds/2799

s390x RHEL 3.x: fail twice
https://buildbot.python.org/all/#/builders/21/builds/2780

AMD64 Debian root 3.x: fail twice
https://buildbot.python.org/all/#/builders/27/builds/2759

[vstinner]

New changeset 99fcc61 by Victor Stinner in branch 'master':
Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" (GH613006)
99fcc61

[vstinner]

I just reverted the change. I'm unable to reproduce and I am not available this week to investigate anyway.

[1st1]

New changeset 321def8 by Yury Selivanov (Ben Harper) in branch 'master':
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
321def8

[miss-islington]

New changeset 13915a3 by Miss Islington (bot) in branch '3.8':
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
13915a3

[vstinner]

Can we close this issue, or is there a remaining known bug?

[ned-deily]

FWIW, the assertion error on macOS reported above in msg338774 does not seem to occur anymore with current HEAD of master and with recent system clang (Apple clang version 11.0.0 (clang-1100.0.33.8)) on macOS 10.14.6.

[btharper]

The default build of python with ASAN builds successfully. Test suite and PGO build are currently blocked, but that can be separated if it makes more sense to handle them individually.

Would it make sense to add an ASAN build to the CI pipeline to make detection automatic going forward?

[vstinner]

The default build of python with ASAN builds successfully. Test suite and PGO build are currently blocked, but that can be separated if it makes more sense to handle them individually.

Yeah, please open one issue per ASAN issue (try to group similar warnings in the same Python issue?).

Would it make sense to add an ASAN build to the CI pipeline to make detection automatic going forward?

Gregory maintains a clang UBSan buildbot:
https://buildbot.python.org/all/#/builders/135

Usually I prefer to setup a buildbot when most issues are fixed.

I close this issue.