SIGSEV in PyErr_SetObject #87792
Comments
|
Hi, I'm dealing with random crashes when using pymongo in Python 3.7.3 in a Debian Buster. This is the python backtrace: Thread 2 (Thread 0x7f9817d95700 (LWP 221)):
Traceback (most recent call first):
File "/usr/local/lib/python3.7/dist-packages/gevent/_threading.py", line 80, in wait
waiter.acquire() # Block on the native lock
File "/usr/local/lib/python3.7/dist-packages/gevent/_threading.py", line 162, in get
self._not_empty.wait()
File "/usr/local/lib/python3.7/dist-packages/gevent/threadpool.py", line 270, in _worker
task = task_queue.get()
File "/usr/local/lib/python3.7/dist-packages/gevent/threadpool.py", line 254, in __trampoline
g.switch()
Thread 1 (Thread 0x7f981fdfd740 (LWP 216)):
Traceback (most recent call first):
<built-in method decode_all of module object at remote 0x7f981d41ca48>
File "/usr/local/lib/python3.7/dist-packages/bson/__init__.py", line 1089, in _decode_all_selective
return decode_all(data, codec_options)
File "/usr/local/lib/python3.7/dist-packages/pymongo/message.py", line 1616, in unpack_response
self.payload_document, codec_options, user_fields)
File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1080, in _unpack_response
legacy_response)
File "/usr/local/lib/python3.7/dist-packages/pymongo/server.py", line 131, in run_operation_with_response
user_fields=user_fields)
File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1366, in _cmd
unpack_res)
File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
return func(session, server, sock_info, slave_ok)
File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
exhaust=exhaust)
File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1001, in __send_message
address=self.__address)
File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1124, in _refresh
self.__send_message(q)
File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1207, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 1319, in find_one
for result in cursor.limit(-1):
File "/usr/local/lib/python3.7/dist-packages/gecoscc/userdb.py", line 119, in create_user
user = self.collection.find_one({'email': email})
File "/usr/local/lib/python3.7/dist-packages/gecoscc/commands/create_adminuser.py", line 95, in command
{'is_superuser': self.options.is_superuser}
File "/usr/local/lib/python3.7/dist-packages/gecoscc/management.py", line 90, in __call__
self.command()
File "/usr/local/lib/python3.7/dist-packages/gecoscc/management.py", line 48, in main
command()
File "/usr/local/bin/pmanage", line 10, in <module>
sys.exit(main())
(gdb) And this is the builtin-code backtrace: Core was generated by `/usr/bin/python3 /usr/local/bin/pmanage /opt/gecosccui/gecoscc.ini create_admin'. As I understand the code is using "getattr" to ckeck if a dict contains an attribute called "_type_marker", and somehow when Python is formatting the exception finds that the stack has been corrupted. What can be happening? How can I help to debug this? Best regards! |
System225
mannequin
added
3.7 (EOL)
Try to run your application in the Python Debug Mode: https://docs.python.org/dev/library/devmode.html The best is if you can run your application with a Python built in debug mode. Usually, it's a bug in a 3rd party C extension. |
|
Thank you Victor for your response. But, I compiled Python 3.7 from source and modified the "Python/errors.c" code where the problem is detected in the following way: void
PyErr_SetObject(PyObject *exception, PyObject *value)
{
PyThreadState *tstate = PyThreadState_GET();
PyObject *exc_value;
PyObject *tb = NULL;
_PyErr_StackItem *exc_info = NULL; Py_XINCREF(value);
exc_info = _PyErr_GetTopmostException(tstate);
exc_value = exc_info->exc_value;
if (exc_value != NULL && exc_value != Py_None) {
/* Implicit exception chaining */
printf("exc_value=%p\n", exc_value);
printf("exc_info=%p\n", exc_info);
printf("tstate=%p\n", tstate);
printf("traceback=%p\n", exc_info->exc_traceback);
printf("exc_value.ob_type=%p\n", exc_value->ob_type);
Py_INCREF(exc_value);
In this way I had the pointer printed just before the error happening:exc_value=0x73726573756e69
exc_info=0x7f83bfafacb8
tstate=0x5605dcd41330
traceback=0x6b6f0100
Segmentation fault (core dumped)And by using gdb I printed the memory contents: (gdb) x/32c 0x7f83bfafacb8 If you check the original post you will see: #15 0x000000000053f732 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x17882e8, for file /usr/local/lib/python3.7/dist-packages/bson/__init__.py, line 1013, in decode_all (data=b'V\x00\x00\x00\x03cursor\x00=\x00\x00\x00\x04firstBatch\x00\x05\x00\x00\x00\x00\x12id\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02ns\x00\x13\x00\x00\x00gecoscc.adminusers\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\xf0?\x00', codec_options=<CodecOptions at remote 0x7f1ab4ac2208>, view=<memoryview at remote 0x7f1ab5375708>, data_len=86, docs=[], position=0, end=85)) at ../Python/ceval.c:547So, somehow the "oscc.adminusers\x00\x00\x01ok\x00" part of the message is written over the exc_info memory. That makes me think that this is a problem in pymongo module. I will report this bug to them. Thank you very much! |
Yeah, in 90% of cases, the bug comes from a third party C extensions. That's why Python 3.10 now dumps the list of 3rd party C extensions on a fatal error :-) https://twitter.com/VictorStinner/status/1374736944022876169 |
|
This issue was resolved in https://jira.mongodb.org/browse/PYTHON-2621 The cause of the segfault was determined to be gevent 1.3.4 (2018) and/or greenlet 0.4.13 (2018). When the reporter upgraded to gevent==21.1.2 and greenlet==1.0 the segfault went away. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: