What causes double free or corruption (!prev):?

[brizzbane]

I have built a custom pycurl client that runs on top of pyuv. Am also using pycares.

I just started using pycares w/my client, and have run into this issue consistently:

*** Error in `python': double free or corruption (!prev): 0x0000000002cf6280 ***
Aborted

From the little I could find online. My 'guess', is that maybe its a problem with trying to run the loop once for a domain that might not have MX records? The code ran fine, then once it got to certain records (email addresses)., the error happens consistently. Here is my code:

    for itm in response:
        user, domain = itm._id.split('@')
        print(user)

        def handle_response(future):
            response = future.result()
            print('response %s' % response)
            print('code %s' % (response.code,))
            itm.smtp_code = response.code
            itm.save()
            self.client._set_timeout(0)

        def query_cb(result, error):
            if result is not None:
                r = cURLSMTPRequest('smtp://%s' % result[0].host,
                                                mail_from='test@test.com',
                                                mail_rcpt=itm._id,
                                                message='hi')


                itm.activity.last.attempt = datetime.now()
                itm.save()
                future = self.client.req(r)
                future.add_done_callback(handle_response, future)
                ioloop.run(pyuv.UV_RUN_ONCE)

        self.resolve.query(domain, pycares.QUERY_TYPE_MX, query_cb)
        self.client.ioloop.run(pyuv.UV_RUN_ONCE)

Is this a 'bug', or is there a way to resolve the problem? I've spent a looong time getting a nice event loop going w/pycurl where CPU issues are non existant. Any help you can point me to is much appreciated!

[saghul]

Do you have a backtrace? There are too many moving parts in your example, it's very hard to know where the error comes from.

[brizzbane]

Started looking at how to do that, but not sure what I'm needing to do. Familiar w/cProfile. ..is there something like python -m backtrace -o debug.dump programthatcrashes.py?

..Googling now. I thought thats what I needed last night, but wasn't sure.

[saghul]

If you are using LInux you can do the following:

ulimit -c unlimited
gdb python foom.py
gdb> run

Once it crashes, get a backtrace using "bt" in the console.

[brizzbane]

ok here is is (and thanks for explanation on how to get!)

#0 0x00007ffff6f26507 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007ffff6f278da in __GI_abort () at abort.c:89
#2 0x00007ffff6f64a63 in __libc_message (do_abort=do_abort@entry=1,
fmt=fmt@entry=0x7ffff705d838 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6f69ebe in malloc_printerr (action=1, str=0x7ffff705d8b0 "double free or corruption (!prev)", ptr=)
at malloc.c:4965
#4 0x00007ffff6f6a69b in _int_free (av=, p=, have_lock=0) at malloc.c:3834
#5 0x00007ffff6702c64 in ares__free_query (query=0x20d8700) at src/ares_process.c:1316
#6 0x00007ffff6702b56 in end_query (channel=0xeb9180, query=0x20d8700, status=0, abuf=0x7fffffff4420 "gy\201\200", alen=54)
at src/ares_process.c:1290
#7 0x00007ffff6701770 in process_answer (channel=0xeb9180, abuf=0x7fffffff4420 "gy\201\200", alen=54, whichserver=0, tcp=0,
now=0x7fffffff5470) at src/ares_process.c:613
#8 0x00007ffff6701300 in read_udp_packets (channel=0xeb9180, read_fds=0x0, read_fd=16, now=0x7fffffff5470)
at src/ares_process.c:478
#9 0x00007ffff6700808 in processfds (channel=0xeb9180, read_fds=0x0, read_fd=16, write_fds=0x0, write_fd=-1)
at src/ares_process.c:127
#10 0x00007ffff670089b in ares_process_fd (channel=0xeb9180, read_fd=16, write_fd=-1) at src/ares_process.c:148
#11 0x00007ffff66f6243 in Channel_func_process_fd (self=0x7ffff01286c8, args=) at src/cares.c:1147
#12 0x00000000004bbc2a in call_function (oparg=, pp_stack=0x7fffffff5570) at ../Python/ceval.c:4350
#13 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
#14 0x00000000004b9416 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
#15 0x00000000004d5269 in function_call.lto_priv () at ../Objects/funcobject.c:526
#16 0x00000000004edbce in PyObject_Call (kw=0x0, arg=0x7fffef791158, func=0x7ffff40c4a28) at ../Objects/abstract.c:2546
#17 instancemethod_call.lto_priv () at ../Objects/classobject.c:2602
#18 0x00000000004aefc3 in PyObject_Call (kw=0x0, arg=0x7fffef7d90f0, func=0x7fffefa547d0) at ../Objects/abstract.c:2546
#19 PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2773
#20 0x00007ffff6b2371e in pyuv__poll_cb (handle=0x7ffff38253c8, status=0, events=) at src/poll.c:26
#21 0x00007ffff6b4380c in uv__poll_io (loop=0x7ffff6d69180 <default_loop_struct>, w=0x7ffff3825430, events=1) at src/unix/poll.c:49
#22 0x00007ffff6b4e9ed in uv__io_poll (loop=0x7ffff6d69180 <default_loop_struct>, timeout=1) at src/unix/linux-core.c:345
#23 0x00007ffff6b3bbb9 in uv_run (loop=0x7ffff6d69180 <default_loop_struct>, mode=UV_RUN_ONCE) at src/unix/core.c:341
#24 0x00007ffff6b2ecb8 in Loop_func_run (self=0xea8df0, args=) at src/loop.c:62
[edit] and more I guess:

#25 0x00000000004bbc2a in call_function (oparg=, pp_stack=0x7fffffff8d10) at ../Python/ceval.c:4350
#26 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
#27 0x00000000004b9416 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
#28 0x00000000004d5269 in function_call.lto_priv () at ../Objects/funcobject.c:526
#29 0x00000000004aefc3 in PyObject_Call (kw=0x0, arg=0x7ffff0106998, func=0x7fffef78e398) at ../Objects/abstract.c:2546
#30 PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2773
#31 0x00007ffff66f45c1 in query_mx_cb (arg=0x7fffef78e398, status=, timeouts=, answer_buf=, answer_len=) at src/cares.c:309
#32 0x00007ffff6703113 in qcallback (arg=0x20bee40, status=0, timeouts=0, abuf=0x7fffffff9170 "(j\201\200", alen=109) at src/ares_query.c:183
#33 0x00007ffff6702b4a in end_query (channel=0xeb9180, query=0x20e3a40, status=0, abuf=0x7fffffff9170 "(j\201\200", alen=109) at src/ares_process.c:1289
#34 0x00007ffff6701770 in process_answer (channel=0xeb9180, abuf=0x7fffffff9170 "(j\201\200", alen=109, whichserver=0, tcp=0, now=0x7fffffffa1c0) at src/ares_process.c:613
#35 0x00007ffff6701300 in read_udp_packets (channel=0xeb9180, read_fds=0x0, read_fd=16, now=0x7fffffffa1c0) at src/ares_process.c:478
#36 0x00007ffff6700808 in processfds (channel=0xeb9180, read_fds=0x0, read_fd=16, write_fds=0x0, write_fd=-1) at src/ares_process.c:127
#37 0x00007ffff670089b in ares_process_fd (channel=0xeb9180, read_fd=16, write_fd=-1) at src/ares_process.c:148
#38 0x00007ffff66f6243 in Channel_func_process_fd (self=0x7ffff01286c8, args=) at src/cares.c:1147
#39 0x00000000004bbc2a in call_function (oparg=, pp_stack=0x7fffffffa2c0) at ../Python/ceval.c:4350
#40 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
#41 0x00000000004b9416 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
#42 0x00000000004d5269 in function_call.lto_priv () at ../Objects/funcobject.c:526
#43 0x00000000004edbce in PyObject_Call (kw=0x0, arg=0x7ffff01251b0, func=0x7ffff40c4a28) at ../Objects/abstract.c:2546
#44 instancemethod_call.lto_priv () at ../Objects/classobject.c:2602
#45 0x00000000004aefc3 in PyObject_Call (kw=0x0, arg=0x7fffef7d9190, func=0x7fffefa547d0) at ../Objects/abstract.c:2546
#46 PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2773
#47 0x00007ffff6b2371e in pyuv__poll_cb (handle=0x7ffff38253c8, status=0, events=) at src/poll.c:26
#48 0x00007ffff6b4380c in uv__poll_io (loop=0x7ffff6d69180 <default_loop_struct>, w=0x7ffff3825430, events=1) at src/unix/poll.c:49
#49 0x00007ffff6b4e9ed in uv__io_poll (loop=0x7ffff6d69180 <default_loop_struct>, timeout=40) at src/unix/linux-core.c:345
#50 0x00007ffff6b3bbb9 in uv_run (loop=0x7ffff6d69180 <default_loop_struct>, mode=UV_RUN_ONCE) at src/unix/core.c:341
#51 0x00007ffff6b2ecb8 in Loop_func_run (self=0xea8df0, args=) at src/loop.c:62
#52 0x00000000004bbc2a in call_function (oparg=, pp_stack=0x7fffffffda60) at ../Python/ceval.c:4350
#53 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
#54 0x00000000004b9416 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
#55 0x00000000004c16f3 in fast_function (nk=, na=, n=1, pp_stack=0x7fffffffdc60, func=0x7fffefa48578) at ../Python/ceval.c:4446
#56 call_function (oparg=, pp_stack=0x7fffffffdc60) at ../Python/ceval.c:4371
#57 PyEval_EvalFrameEx () at ../Python/ceval.c:2987
#58 0x00000000004b9416 in PyEval_EvalCodeEx () at ../Python/ceval.c:3582
#59 0x00000000004e9f4f in PyEval_EvalCode (locals=0x7ffff7f65168, globals=0x7ffff7f65168, co=0x7ffff7e812b0) at ../Python/ceval.c:669
#60 run_mod.lto_priv () at ../Python/pythonrun.c:1370
#61 0x00000000004e4b42 in PyRun_FileExFlags () at ../Python/pythonrun.c:1356
#62 0x00000000004e3406 in PyRun_SimpleFileExFlags () at ../Python/pythonrun.c:948
#63 0x0000000000492de3 in Py_Main () at ../Modules/main.c:640
#64 0x00007ffff6f13870 in __libc_start_main (main=0x492830

, argc=2, argv=0x7fffffffe0a8, init=, fini=, rtld_fini=, stack_end=0x7fffffffe098) at libc-start.c:291
#65 0x0000000000492759 in _start ()

[saghul]

Seems to be a pycares issue, can you please open the issue there? The problem seems to be caused by parsing the answer, can you please tell me what domain you're trying to resolve?

[saghul]

Closing, continues in saghul/pycares#26