Segfaults on OS X when calling Scraper.register(conn) #4

simonw · 2018-05-25T06:30:26Z

This is really weird. In #3 I reported that everything was working fine for me. Now I'm getting a setfault every time I try to run register(). Here's a sample session:

Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import re, requests, sqlite3
>>> from vtfunc import TableFunction
>>> 
>>> class Scraper(TableFunction):
...     params = ['url']  # Function argument names.
...     columns = ['href', 'description']  # Result rows have these columns.
...     name = 'scraper'  # Name we use to invoke the function from SQL.
...     def initialize(self, url):
...         # Note that this is bytes:
...         self.html = requests.get(url).content
...         self._iter = re.finditer(
...             '<a[^\>]+?href="([^\"]+?)"[^\>]*?>([^\<]+?)</a>',
...             self.html)
...     def iterate(self, idx):
...         return next(self._iter).groups()
... 
>>> conn = sqlite3.connect(':memory:')
>>> Scraper.register(conn)
Segmentation fault: 11

The OS X crash reporter kicks in at this point, which means I can grab the following stack trace:

Process:               Python [44252]
Path:                  /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.6.5 (3.6.5)
Code Type:             X86-64 (Native)
Parent Process:        bash [27024]
Responsible:           Python [44252]
User ID:               502

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [0]

VM Regions Near 0:
--> 
    __TEXT                 000000010e135000-000000010e137000 [    8K] r-x/rwx SM=COW  /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   ???                           	000000000000000000 0 + 0
1   libsqlite3.dylib              	0x00007fff7db348b0 createModule + 80
2   vtfunc.cpython-36m-darwin.so  	0x0000000110115cbc __pyx_f_6vtfunc_18_TableFunctionImpl_create_module + 364
3   vtfunc.cpython-36m-darwin.so  	0x000000011011b189 __pyx_pw_6vtfunc_13TableFunction_1register + 169
4   org.python.python             	0x000000010e143f9c _PyObject_FastCallDict + 143
5   org.python.python             	0x000000010e1e0356 call_function + 441
6   org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
7   org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
8   org.python.python             	0x000000010e1d7f35 PyEval_EvalCodeEx + 57
9   org.python.python             	0x000000010e1641c4 function_call + 339
10  org.python.python             	0x000000010e143e52 PyObject_Call + 101
11  org.python.python             	0x000000010e1d94a4 _PyEval_EvalFrameDefault + 5428
12  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
13  org.python.python             	0x000000010e1e11eb fast_function + 218
14  org.python.python             	0x000000010e1e035d call_function + 448
15  org.python.python             	0x000000010e1d92d4 _PyEval_EvalFrameDefault + 4964
16  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
17  org.python.python             	0x000000010e1e11eb fast_function + 218
18  org.python.python             	0x000000010e1e035d call_function + 448
19  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
20  org.python.python             	0x000000010e1e148b _PyFunction_FastCall + 121
21  org.python.python             	0x000000010e1e035d call_function + 448
22  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
23  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
24  org.python.python             	0x000000010e1e13bb _PyFunction_FastCallDict + 449
25  org.python.python             	0x000000010e143fd1 _PyObject_FastCallDict + 196
26  org.python.python             	0x000000010e1440f4 _PyObject_Call_Prepend + 156
27  org.python.python             	0x000000010e143e52 PyObject_Call + 101
28  org.python.python             	0x000000010e18e510 slot_tp_call + 50
29  org.python.python             	0x000000010e143f9c _PyObject_FastCallDict + 143
30  org.python.python             	0x000000010e144355 _PyObject_FastCallKeywords + 97
31  org.python.python             	0x000000010e1e0356 call_function + 441
32  org.python.python             	0x000000010e1d92d4 _PyEval_EvalFrameDefault + 4964
33  org.python.python             	0x000000010e1e148b _PyFunction_FastCall + 121
34  org.python.python             	0x000000010e1e035d call_function + 448
35  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
36  org.python.python             	0x000000010e1e148b _PyFunction_FastCall + 121
37  org.python.python             	0x000000010e1e035d call_function + 448
38  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
39  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
40  org.python.python             	0x000000010e1d7f35 PyEval_EvalCodeEx + 57
41  org.python.python             	0x000000010e1641c4 function_call + 339
42  org.python.python             	0x000000010e143e52 PyObject_Call + 101
43  org.python.python             	0x000000010e1d94a4 _PyEval_EvalFrameDefault + 5428
44  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
45  org.python.python             	0x000000010e1e13bb _PyFunction_FastCallDict + 449
46  org.python.python             	0x000000010e143fd1 _PyObject_FastCallDict + 196
47  org.python.python             	0x000000010e1440f4 _PyObject_Call_Prepend + 156
48  org.python.python             	0x000000010e143e52 PyObject_Call + 101
49  org.python.python             	0x000000010e1d94a4 _PyEval_EvalFrameDefault + 5428
50  org.python.python             	0x000000010e1e148b _PyFunction_FastCall + 121
51  org.python.python             	0x000000010e1e035d call_function + 448
52  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
53  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
54  org.python.python             	0x000000010e1e11eb fast_function + 218
55  org.python.python             	0x000000010e1e035d call_function + 448
56  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
57  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
58  org.python.python             	0x000000010e1e13bb _PyFunction_FastCallDict + 449
59  org.python.python             	0x000000010e143fd1 _PyObject_FastCallDict + 196
60  org.python.python             	0x000000010e1440f4 _PyObject_Call_Prepend + 156
61  org.python.python             	0x000000010e143e52 PyObject_Call + 101
62  org.python.python             	0x000000010e1d94a4 _PyEval_EvalFrameDefault + 5428
63  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
64  org.python.python             	0x000000010e1e13bb _PyFunction_FastCallDict + 449
65  org.python.python             	0x000000010e143fd1 _PyObject_FastCallDict + 196
66  org.python.python             	0x000000010e1440f4 _PyObject_Call_Prepend + 156
67  org.python.python             	0x000000010e143e52 PyObject_Call + 101
68  org.python.python             	0x000000010e18e510 slot_tp_call + 50
69  org.python.python             	0x000000010e143f9c _PyObject_FastCallDict + 143
70  org.python.python             	0x000000010e1e0356 call_function + 441
71  org.python.python             	0x000000010e1d923b _PyEval_EvalFrameDefault + 4811
72  org.python.python             	0x000000010e1e0ae8 _PyEval_EvalCodeWithName + 1719
73  org.python.python             	0x000000010e1d7ef6 PyEval_EvalCode + 42
74  org.python.python             	0x000000010e200c2a run_mod + 54
75  org.python.python             	0x000000010e1ffc4b PyRun_FileExFlags + 160
76  org.python.python             	0x000000010e1ff328 PyRun_SimpleFileExFlags + 285
77  org.python.python             	0x000000010e213673 Py_Main + 3471
78  org.python.python             	0x000000010e136e1d 0x10e135000 + 7709
79  libdyld.dylib                 	0x00007fff7e22f015 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x00000000a029a697  rbx: 0x00007fb15ae23820  rcx: 0x00007fb15cc44848  rdx: 0x00000001101928d0
  rdi: 0x00007fb15ae23820  rsi: 0x000000010ff6e080  rbp: 0x00007ffee1ac7a00  rsp: 0x00007ffee1ac79c8
   r8: 0x0000000000000000   r9: 0x000000010e28c770  r10: 0x00007ffee1ac790f  r11: 0x00000000012f00e0
  r12: 0x000000010ff6e080  r13: 0x00000001101928d0  r14: 0x0000000000000000  r15: 0x00007fb15ae4b830
  rip: 0x0000000000000000  rfl: 0x0000000000010202  cr2: 0x0000000000000000
  
Logical CPU:     4
Error Code:      0x00000014
Trap Number:     14

The text was updated successfully, but these errors were encountered:

coleifer · 2018-05-25T14:03:43Z

In #3 I reported that everything was working fine for me. Now I'm getting a setfault every time I try to run register().

What has changed? What's different between then and now? I would expect any such problem to be deterministic. In other words, I would expect something important must have changed between when you had it working and now.

coleifer · 2018-05-25T14:25:15Z

From the stack-trace it looks like sqlite3_create_module() is getting passed a null pointer, but I'm not clear what might be causing that.

simonw · 2018-05-25T14:44:32Z

This is the thing that baffles me: I'm reasonably sure I'm running the exact same sequence of commands that I described in #2 and #3 but I'm now getting a different result. I don't think I've upgraded my system Python or anything like that. It's really weird.

I notice that in #2 (comment) I installed the module and checked that it could be imported but did not attempt to use it. But I'm certain I saw the scraper example work with 0.3.3 yesterday (though I don't think I recorded a full Python shell transcript). And now I can't get it to work at all.

At least this time I've been able to get a stacktrace out of it. I wish I could answer your question about "what has changed" - because obviously something has, but I'm at a loss for what it is.

simonw · 2018-05-25T14:46:38Z

Here's my current full steps-to-reproduce:

$ python3 -mvenv vtfunc-0-3-3
$ source vtfunc-0-3-3/bin/activate
$ pip install vtfunc
$ pip install requests
$ python

import re, sqlite3
from vtfunc import TableFunction

class Scraper(TableFunction):
    params = ['url']  # Function argument names.
    columns = ['href', 'description']  # Result rows have these columns.
    name = 'scraper'  # Name we use to invoke the function from SQL.
    def initialize(self, url):
        # Note that this is bytes:
        self.html = requests.get(url).content
        self._iter = re.finditer(
            '<a[^\>]+?href="([^\"]+?)"[^\>]*?>([^\<]+?)</a>',
            self.html)
    def iterate(self, idx):
        return next(self._iter).groups()

conn = sqlite3.connect(':memory:')
Scraper.register(conn)

This dies with Segmentation fault: 11

simonw · 2018-05-25T14:49:53Z

Huh... I appear to have fixed it! I used the above steps but I swapped out pip install vtfunc for this:

pip install --no-cache-dir vtfunc

The above example no longer segfaults.

simonw closed this as completed May 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfaults on OS X when calling Scraper.register(conn) #4

Segfaults on OS X when calling Scraper.register(conn) #4

simonw commented May 25, 2018

coleifer commented May 25, 2018

coleifer commented May 25, 2018

simonw commented May 25, 2018 •

edited

Loading

simonw commented May 25, 2018

simonw commented May 25, 2018

Segfaults on OS X when calling Scraper.register(conn) #4

Segfaults on OS X when calling Scraper.register(conn) #4

Comments

simonw commented May 25, 2018

coleifer commented May 25, 2018

coleifer commented May 25, 2018

simonw commented May 25, 2018 • edited Loading

simonw commented May 25, 2018

simonw commented May 25, 2018

simonw commented May 25, 2018 •

edited

Loading