int(s), float(s) and others may cause segmentation fault #78268
Comments
|
fenrrir
mannequin
added
3.7 (EOL)
|
It's difficult to investigate without reproducible example. Do you use 3rd party extension module? |
|
Sorry for that. I’m not able to make a little example to reproduce this bug. It’s happens during Django tests on a very large code base. A interest thing is that not happens with “-X dev” parameters. Please, any suggestions how I can get more details about this bug? |
|
If example cannot be small, you can share the project on Github or other place. 3rd party library list may helpful, but I'm not sure. |
|
Running with Fatal Python error: Segmentation fault Current thread 0x00007fff89cf2380 (most recent call first): The Django code: def resolve(self, path):
path = str(path) # path may be a reverse_lazy object
tried = []
match = self.pattern.match(path)
if match:
new_path, args, kwargs = match
for pattern in self.url_patterns:
try:
sub_match = pattern.resolve(new_path)
except Resolver404 as e:
sub_tried = e.args[0].get('tried')
if sub_tried is not None:
tried.extend([pattern] + t for t in sub_tried)
else:
tried.append([pattern])
else:
if sub_match:
# Merge captured arguments in match with submatch
sub_match_dict = dict(kwargs, **self.default_kwargs)
# Update the sub_match_dict with the kwargs from the sub_match.
sub_match_dict.update(sub_match.kwargs)
# If there are *any* named groups, ignore all non-named groups.
# Otherwise, pass all non-named arguments as positional arguments.
sub_match_args = sub_match.args
if not sub_match_dict:
sub_match_args = args + sub_match.args
return ResolverMatch(
sub_match.func,
sub_match_args,
sub_match_dict,
sub_match.url_name,
[self.app_name] + sub_match.app_names,
[self.namespace] + sub_match.namespaces,
)
tried.append([pattern]) # Segfault happens here!
raise Resolver404({'tried': tried, 'path': new_path})
raise Resolver404({'path': path})According to faulthandler the segmentation fault happens on tried.append([pattern]). |
|
Please try to set PYTHONMALLOC=debug environment variable and run again your test. I hope that Python will be able to catch a memory corruption somewhere. If it's the case, you can try: PYTHONMALLOC=debug python3 -X tracemalloc=10 (...)To see where the corrupted object has been allocated. |
vstinner
changed the title
|
Unfortunately with 'PYTHONMALLOC=debug' the segmentation fault do not happen. |
|
Would you provide list of extension modules? |
|
I also have a segfault that goes away with malloc debugging. Not sure if it's the same issue. My extension modules are venv/lib/python3.7/site-packages//_yaml.cpython-37m-darwin.so Unfortunately I can't test the application without any of these, but maybe we can match my list with fenrrir's. |
|
Btw my segfault is from Django too, but that may just be a coincidence |
|
Extensions: ./_cffi_backend.cpython-37m-darwin.so requirements.txt alabaster==0.7.11 |
|
Hmm, all segfault are happend on macOS? |
|
I can reproduce this on Ubuntu 18.04. INADA, I have a full gdb backtrace with Python 3.7 development build. I'd like to share it with you privately as I'm concerned it may contain sensible information. I know that's a bit unconventional; if you have other suggestions I'm happy to follow along. |
|
I did remove PyYAML, lxml and Pillow here but segfault still happen |
|
@jonasH, Thank you. Would you share core file too? My mail address is "songofacandy [at] gmail.com". |
|
I received two core files ("core" and "core.23") from fenrrir, thank you. In core.23, SEGV happens in kill() syscall. I can't get how it can happened. (gdb) bt 10 In "core", SEGV happens in pymalloc. Pool is broken and I suppose some buffer overflow breaks the pool, but I'm not sure. #0 pymalloc_alloc (ptr_p=ptr_p@entry=0x7fffee81a640, nbytes=40, ctx=<optimized out>) at ../Objects/obmalloc.c:1398
#1 0x000055555562ed78 in pymalloc_alloc (ctx=<optimized out>, nbytes=40, ptr_p=0x7fffee81a640) at ../Objects/obmalloc.c:1565
#2 _PyObject_Malloc (ctx=<optimized out>, nbytes=40) at ../Objects/obmalloc.c:1555
#3 0x000055555561d5d5 in new_dict_with_shared_keys (keys=0x7ffff4e4c2d0) at ../Objects/dictobject.c:607
#4 _PyObjectDict_SetItem (tp=tp@entry=0x555555e8b338, dictptr=0x7fffedfe4250, key=key@entry='s', value=value@entry='\n <h3>') at ../Objects/dictobject.c:4311
#5 0x000055555562b2d4 in _PyObject_GenericSetAttrWithDict (dict=0x0, value='\n <h3>', name='s', obj=<TextNode at remote 0x7fffedfe4240>) at ../Objects/object.c:1319
#6 PyObject_GenericSetAttr (obj=<TextNode at remote 0x7fffedfe4240>, name='s', value='\n <h3>') at ../Objects/object.c:1341
#7 0x000055555562a09f in PyObject_SetAttr (v=v@entry=<TextNode at remote 0x7fffedfe4240>, name=<optimized out>, value=value@entry='\n <h3>') at ../Objects/object.c:978
#8 0x00005555555c67d8 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:2002
#9 0x00005555555c1787 in function_code_fastcall (co=co@entry=0x7ffff4e90930, args=<optimized out>, args@entry=0x7fffee81a930, nargs=2,
globals=globals@entry={'__name__': 'django.template.base', '__doc__': "\nThis is the Django template system...
../Objects/call.c:283(gdb) p pool |
methane
changed the title
|
Reduced it to something that seems unicode related? No extension modules involved. Vanilla Django project with a single url + template. See testproj/urls.py and tmpl/index.html |
|
If someone expects progress on these issues, we need to get a repository that we can clone to reproduce the bug. Then the code must be simplified as much as possible to reduce the code base. The best is when you still trigger the crash with less than 1000 lines of code and no or very few dependencies. |
|
@jonasH Thanks! |
|
Sure. Unpack archive, create new 3.7 venv with Django (latest version is fine), ./manage.py runserver, curl localhost:8000. |
|
I tried it but I can't reproduce... |
|
@jonasH What Python binary do you use?
|
|
Here's a Docker image that reproduces the bug. FROM ubuntu:18.04 Of course this also works outside Docker. I have reproduced with macOS 10.13.5 (17F77) and Ubuntu 18.04 (Docker). On macOS it's Python 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)] on darwin, installed from Homebrew. On Ubuntu it's whatever is in the 18.04 repositories. |
|
I can reproduce the segmentation fault using 'testproj.tar.gz' with homebrew and compiled from source. MacOS X High Sierra 10.13.5 (17F77). |
|
Very thanks! I found I set Now I can reproduce and I get where memory is broken, and understood what cause: Python traceback is:
Traceback (most recent call first):
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 767, in __init__
self.literal = float(var)
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 646, in __init__
var_obj = Variable(constant).resolve({})
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 568, in compile_filter
return FilterExpression(token, self)
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/templatetags/i18n.py", line 368, in do_translate
message_string = parser.compile_filter(bits[1])
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse
compiled_result = compile_func(self, token)
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loader_tags.py", line 233, in do_block
nodelist = parser.parse(('endblock',))
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse
compiled_result = compile_func(self, token)
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 198, in compile_nodelist
return parser.parse()
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 160, in __init__
self.nodelist = self.compile_nodelist()
File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loaders/base.py", line 30, in get_templateC traceback is: #0 0x0000557880378c95 in _Py_string_to_number_with_underscores (s=0x7ff07d8e7bc0 "\"So erstellen Sie einen Zahlungslink f?ded_blocks'", orig_len=39, what=0x55788046d60a "float",
obj='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"', arg=0x7ff07d9368b0, innerfunc=0x557880276b10 <float_from_string_inner>) at Python/pystrtod.c:410
#1 0x0000557880279b23 in PyFloat_FromString (v='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"') at Objects/floatobject.c:208
#2 0x0000557880279cd5 in float_new_impl (x=<optimized out>, type=0x5578806e4ee0 <PyFloat_Type>) at Objects/floatobject.c:1627
#3 float_new (type=<optimized out>, args=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwargs=<optimized out>) at Objects/clinic/floatobject.c.h:181
#4 0x00005578802cfdc5 in type_call (type=type@entry=0x5578806e4ee0 <PyFloat_Type>, args=args@entry=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwds=kwds@entry=0x0)
at Objects/typeobject.c:929
#5 0x000055788026b3bf in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x5578806e4ee0>, stack=<optimized out>, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0)
at Objects/call.c:199
#6 0x00005578802526f4 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4600
#7 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3183
#8 0x000055788024fb17 in function_code_fastcall (co=co@entry=0x7ff07ec4f0c0, args=<optimized out>, args@entry=0x7ffc786f2330, nargs=2,
globals=globals@entry={'__name__': 'django.template.base', '__doc__': "\nThis is the Django template system.\n\nHow it works:\n\nThe Lexer.tokenize() function converts a template string
(i.e., a string containing\nmarkup with custom template tags) to tokens, which can be either plain text\n(TOKEN_TEXT), variables (TOKEN_VAR) or block statements (TOKEN_BLOCK).\n\nThe Parser(
) class takes a list of tokens in its constructor, and its parse()\nmethod returns a compiled template -- which is, under the hood, a list of\nNode objects.\n\nEach Node is responsible for c
reating some sort of output -- e.g. simple text\n(TextNode), variable values in a given context (VariableNode), results of basic\nlogic (IfNode), results of looping (ForNode), or anything el
se. The core Node\ntypes are TextNode, VariableNode, IfNode and ForNode, but plugin modules can\ndefine their own custom node types.\n\nEach Node has a render() method, which takes a Context
and returns a string of\nthe rendered node. For example, the render() method of a Variable Node ret...(truncated)) at Objects/call.c:283
#9 0x000055788026ad87 in _PyFunction_FastCallDict (func=<function at remote 0x7ff07ec2f1e0>, args=0x7ffc786f2330, nargs=<optimized out>, kwargs=0x0) at Objects/call.c:322
#10 0x000055788026ddc1 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffc786f2330, callable=<function at remote 0x7ff07ec2f1e0>) at Objects/call.c:98 |
methane
changed the title
|
It might be a duplicate of bpo-33954: bug in _PyUnicode_InsertThousandsGrouping(). |
methane
changed the title
|
Is it possible to write a reproducer in Python? The string returned by _PyUnicode_TransformDecimalAndSpaceToASCII() is still a NUL-terminated, but the NUL can be far past the end of the string. |
|
I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes. |
We just have to call float() with a non-ASCII string, no? An assertion must fail without the fix when Python is compiled in debug mode. |
|
The assertion in the patched code, yes. The segfault in the unpatched code, no. |
|
When I commented out the fix: $ ./python -c 'int("こんにちは")'
python: Objects/unicodeobject.c:484: _PyUnicode_CheckConsistency: Assertion `PyUnicode_READ(kind, data, ascii->length) == 0' failed.
AbortedI added this to test cases. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: