Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash with CURLSHOPT_SHARE / CURL_LOCK_DATA_CONNECT #2219

Closed
prapin opened this issue Jan 5, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@prapin
Copy link

commented Jan 5, 2018

I did this

I enabled the shared connection pooling feature (CURL_LOCK_DATA_CONNECT), newly introduced in 7.57.0. I used a C++ std::recursive_mutex to implement locking for shared object.
My application is heavily multi-threaded. At the time of the crash, 10 identical threads but working
on different data sets could possibly perform HTTP requests simultaneously, and to the same server.

My IDE if Xcode 9.2 and I enabled a powerful feature called Address Sanitizer.
This is like Valgrind on Linux, but much faster and well integrated in the IDE.
AddressSanitizer detects a bug, use of heap memory being already freed, in conncache.c.

The stack trace in the crashing thread is like:

frame #5: 0x0000000101592882 bundle_add_conn(cb_ptr=0x00006040004abc50, conn=0x000061d002860480) at conncache.c:87
frame #6: 0x000000010159207e Curl_conncache_add_conn(connc=0x000061100009ade0, conn=0x000061d002860480) at conncache.c:248
frame #7: 0x000000010171a0d2 create_conn(data=0x000062a00018c200, in_connect=0x000062a00018c210, async=0x000070000c2b9710) at url.c:4526
frame #8: 0x0000000101714dfe Curl_connect(data=0x000062a00018c200, in_connect=0x000062a00018c210, asyncp=0x000070000c2b9710, protocol_done=0x000070000c2b9720) at url.c:4680
frame #9: 0x0000000101667277 multi_runsingle(multi=0x00006150001b0600, now=(tv_sec = 1515153624, tv_usec = 548658), data=0x000062a00018c200) at multi.c:1432
frame #10: 0x00000001016654df curl_multi_perform(multi=0x00006150001b0600, running_handles=0x000070000c2ba6c0) at multi.c:2162
frame #11: 0x00000001015c3ffa easy_transfer(multi=0x00006150001b0600) at easy.c:683
frame #12: 0x00000001015c0ad3 easy_perform(data=0x000062a00018c200, events=false) at easy.c:769
frame #13: 0x00000001015c07f7 curl_easy_perform(data=0x000062a00018c200) at easy.c:788

AddressSanitizer reports that cb_ptr variable in bundle_add_conn function, as well as
bundle and new_bundle variables in Curl_conncache_add_conn are all
pointing to invalid addresses (already destroyed, displayed in yellow on Xcode).

At the crashing time, 7 threads were blocked while waiting for the mutex (proving that locking
in working as expected) with a stack trace like below. No other thread is dealing with cURL at that time.

frame #4: 0x000000010183743a (anonymous namespace)::CurlShared::lock(handle=0x000062a0001aa200, data=CURL_LOCK_DATA_CONNECT, access=CURL_LOCK_ACCESS_SINGLE, userptr=0x0000614000040640) at BaseCurlHttpRequest.cpp:49
frame #5: 0x00000001016c4e98 Curl_share_lock(data=0x000062a0001aa200, type=CURL_LOCK_DATA_CONNECT, accesstype=CURL_LOCK_ACCESS_SINGLE) at share.c:227
frame #6: 0x000000010159163b Curl_conncache_find_bundle(conn=0x000061d002850a80, connc=0x000061100009ade0) at conncache.c:177
frame #7: 0x0000000101727e15 ConnectionExists(data=0x000062a0001aa200, needle=0x000061d002850a80, usethis=0x000070000cb6ba60, force_reuse=0x000070000cb6baf0, waitpipe=0x000070000cb6bb00) at url.c:1104
frame #8: 0x00000001017191b3 create_conn(data=0x000062a0001aa200, in_connect=0x000062a0001aa210, async=0x000070000cb6c710) at url.c:4411
frame #9: 0x0000000101714dfe Curl_connect(data=0x000062a0001aa200, in_connect=0x000062a0001aa210, asyncp=0x000070000cb6c710, protocol_done=0x000070000cb6c720) at url.c:4680
frame #10: 0x0000000101667277 multi_runsingle(multi=0x00006150001e0800, now=(tv_sec = 1515153624, tv_usec = 550299), data=0x000062a0001aa200) at multi.c:1432
frame #11: 0x00000001016654df curl_multi_perform(multi=0x00006150001e0800, running_handles=0x000070000cb6d6c0) at multi.c:2162
frame #12: 0x00000001015c3ffa easy_transfer(multi=0x00006150001e0800) at easy.c:683
frame #13: 0x00000001015c0ad3 easy_perform(data=0x000062a0001aa200, events=false) at easy.c:769
frame #14: 0x00000001015c07f7 curl_easy_perform(data=0x000062a0001aa200) at easy.c:788

And here is the complete report from AddressSanitizer:

SUMMARY: AddressSanitizer: heap-use-after-free conncache.c:87 in bundle_add_conn
 Shadow bytes around the buggy address:
   0x1c0800095730: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 fa
   0x1c0800095740: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 00
   0x1c0800095750: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
   0x1c0800095760: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
   0x1c0800095770: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
 =>0x1c0800095780: fa fa 00 00 00 00 00 00 fa fa fd fd fd[fd]fd fd
   0x1c0800095790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x1c08000957a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x1c08000957b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x1c08000957c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x1c08000957d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 Shadow byte legend (one shadow byte represents 8 application bytes):
   Addressable:           00
   Partially addressable: 01 02 03 04 05 06 07
   Heap left redzone:       fa
   Freed heap region:       fd
   Stack left redzone:      f1
   Stack mid redzone:       f2
   Stack right redzone:     f3
   Stack after return:      f5
   Stack use after scope:   f8
   Global redzone:          f9
   Global init order:       f6
   Poisoned by user:        f7
   Container overflow:      fc
   Array cookie:            ac
   Intra object redzone:    bb
   ASan internal:           fe
   Left alloca redzone:     ca
   Right alloca redzone:    cb

I expected the following

AddressSanitizer should not detect any problem.

curl/libcurl version

7.57.0

operating system

macOS 10.12.6

@jay jay added build crash and removed build labels Jan 6, 2018

@jay

This comment has been minimized.

Copy link
Member

commented Jan 6, 2018

Likely fixed in #2132. Please try reproducing with master.

@prapin

This comment has been minimized.

Copy link
Author

commented Jan 9, 2018

I confirm that master branch (commit 2a6dbb8) does not yield the problem.

@prapin prapin closed this Jan 9, 2018

@lock lock bot locked as resolved and limited conversation to collaborators May 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.