enable usage of AddressSanitizer in CPython [PATCH]

[halfie]

BPO	18596
Nosy	@ncoghlan, @vstinner, @tiran
Files	enable-ASAN-in-CPython.diff: patch to enable ASAN in CPython ASAN-compat-35da5d848ffd-v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-09-28.14:29:15.514>
created_at = <Date 2013-07-30.08:52:39.258>
labels = ['interpreter-core', 'type-feature']
title = 'enable usage of AddressSanitizer in CPython [PATCH]'
updated_at = <Date 2013-09-28.14:31:00.649>
user = 'https://bugs.python.org/halfie'

bugs.python.org fields:

activity = <Date 2013-09-28.14:31:00.649>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2013-09-28.14:29:15.514>
closer = 'python-dev'
components = ['Interpreter Core']
creation = <Date 2013-07-30.08:52:39.258>
creator = 'halfie'
dependencies = []
files = ['31083', '31866']
hgrepos = []
issue_num = 18596
keywords = ['patch']
message_count = 7.0
messages = ['193917', '193935', '193973', '193974', '198382', '198517', '198519']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'christian.heimes', 'neologix', 'python-dev', 'halfie']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18596'
versions = ['Python 3.4']

[halfie]

Hi,

When trying to build CPython "tip" with AddressSanitizer enabled, I get the following crash,

make Parser/pgen
make[1]: Entering directory /scratch/repos/cpython' make[1]: Parser/pgen' is up to date.
make[1]: Leaving directory `/scratch/repos/cpython'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
=================================================================
==1375== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60380000f020 at pc 0x40b260 bp 0x7fff6e6cbbf0 sp 0x7fff6e6cbbe8
READ of size 4 at 0x60380000f020 thread T0
#0 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
#1 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
#2 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
#3 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
#4 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
#5 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
#6 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
#7 0x3cb2221b74 in ?? ??:0
#8 0x401994 in _start ??:?
0x60380000f020 is located 96 bytes to the left of 0-byte region [0x60380000f080,0x60380000f080)
==1375== AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_allocator2.cc:216 "((id)) != (0)" (0x0, 0x0)
#0 0x3ab681237d in ?? ??:0
#1 0x3ab68191b3 in ?? ??:0
#2 0x3ab680563b in ?? ??:0
#3 0x3ab6816e8f in ?? ??:0
#4 0x3ab6817ff1 in ?? ??:0
#5 0x3ab6812763 in ?? ??:0
#6 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
#7 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
#8 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
#9 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
#10 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
#11 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
#12 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
#13 0x3cb2221b74 in ?? ??:0
#14 0x401994 in _start ??:?
make: *** [Include/graminit.h] Error 1

...

AddressSanitizer is a fast memory error detector. See http://clang.llvm.org/docs/AddressSanitizer.html for details.

Build options used
==================

export CFLAGS="-fsanitize=address -g"

export LDFLAGS="-fsanitize=address"

./configure --prefix=/opt/python --with-valgrind

Software Versions
==================

$ hg log

changeset: 84902:6e1dd1ce95b8
branch: 2.7
tag: tip
parent: 84893:1dbcb0299088
user: Terry Jan Reedy <tjreedy@udel.edu>
date: Tue Jul 30 01:37:36 2013 -0400
summary: Make all idle test case names end with 'Test'.

$ gcc --version
gcc (GCC) 4.8.1 20130603 (Red Hat 4.8.1-1)

$ cat /etc/redhat-release 
Fedora release 19 (Schrödinger’s Cat)

Fix / Work-around
=================

See http://code.google.com/p/address-sanitizer/wiki/AddressSanitizer#Turning_off_instrumentation

Turning off the instrumentation for _PyObject_Free and _PyObject_Realloc is required.

Advantages
==========

AddressSanitizer can *now* be used to test and debug CPython.

AddressSanitizer has helped me find a large number of bugs in other softwares.

[halfie]

This patch should only be applied if we are sure that the allocator is fine and doing such things by design.

[neologix]

The warning is due to the Py_ADDRESS_IN_RANGE() macro: it's a know limitation, we have the same problem with valgrind.

This would be a nice neature. It would IMO be even nicer to have an ASAN-enabled buildbot.

[halfie]

Using,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis)) __attribute__ ((noinline))

instead of,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis))

seems to be a more future-proof (against inlining) solution.

[halfie]

I am attaching the latest tested patch against tip.

[python-dev]

New changeset f6792f734fcc by Nick Coghlan in branch 'default':
Close bpo-18596: Support address sanity checking in clang/GCC
http://hg.python.org/cpython/rev/f6792f734fcc

[ncoghlan]

Incorporated, but as Charles-François noted, a buildbot running with it enabled would be nice.