CURL_LOCK_DATA_CONNECT can cause concurrent access to a connection on multiple threads

[c29reid]

I did this

We've seen race conditions when using CURL_LOCK_DATA_CONNECT in libcurl where sometimes two different threads using two different easy handles ends up sharing the same connection pointer at the same time.
This causes crashes when both threads are doing work on the same connection pointer.

I added curl cpp code which stresses CURL_LOCK_DATA_CONNECT and should eventually trigger an ASAN error or crash with curl compiled using clang's address sanitizers.
It's not consistent how it fails since it's a threading issue. I've found that it's more consistent after adding a random sleep after the unlock here https://github.com/curl/curl/blob/master/lib/url.c#L1372.

Sample program which can reproduce this issue:
curl.txt

An example of ASAN output when this issue is hit with additional logging.
asan-output.txt

Notably three threads with different easy handles decide to reuse the 0x61b000fbd688 connection at the same time.

[thread: 825eb700] url.c:1370 chosen connection=0x61b000fbd688
[thread: 825eb700] reuse_conn(old_conn=0x61b000979188, conn=0x61b000fbd688)
[thread: 7bdde700] url.c:1370 chosen connection=0x61b000fbd688
[thread: 7bdde700][thread: 8a7ff700] Cur reuse_conn(old_conn=0x61b0012e1888, conn=0x61b000fbl.c:1370 chosen connecd688)
[thread: 877f9700] url_attach_connnection(data = 0x6230002c4d08,  conn = 0x61b000f1a388)
tion=0x61b000fbd688
[thread: 877f9700] reuse_conn(old_conn=0x61b000a94988, conn=0x61b000fbd688)
[thread: 825eb700] Curl_attach_connnection(data = 0x62300030e508,  conn = 0x61b000fbd688)
[thread: 7bdde700] Curl_attach_connnection(data = 0x6230000e3908,  conn = 0x61b000fbd688)
[thread: 877f9700] Curl_attach_connnection(data = 0x62300021cd08,  conn = 0x61b000fbd688)

curl/libcurl version

This issue is reproducible on libcurl master.

operating system

Ubuntu

[bagder]

I can reproduce and I have a patch in progress. It currently seem to fix the issue partly but not completely and I'm starting to get a clue regarding what's missing. It will probably take me a few more days to present a fix.

[bagder]

This patch seems to have removed the crashes from running the trigger program...

[c29reid]

Awesome, it's looking much more stable with those changes. I haven't seen any ASAN failure or crash yet with the repro code stressing it.

[bagder]

It is certainly a step in the right direction. I'll test the PR a bit more before I land it, since I got some suspicious-looking freebsd crashes first that now don't occur... I don't think I'll have the nerve to merge it before the Nov 6 release anyway!