Repository navigation
sometimes transfers timeout earlier than CURLOPT_CONNECTTIMEOUT #3602
Description
I did this
used curl_multi interface in php to test about 400 proxies at the same time with a CURLOPT_CONNECTTIMEOUT set 30, however sometimes the later requests (about 150 from 400) all outputs "Connection time-out" and the total_time is very small (about 0.002 s)!
this is a question on stackoverflow.com
and this is a php class that tests this behaviour CurlMultiTimeoutBugTester
This is a sample output from the tester, please notice the later requests
I expected the following
every request takes his chance and tries to connect for 30 seconds then outputs "Connection timed out after 30005 milliseconds" and total_time is 30 seconds or succeed
curl/libcurl version
7.59.0 and 7.64.0
operating system
Linux CentOS and Linux fedora
Activity
[-]sometimes later requests in curl_multi timeout very small than CURLOPT_CONNECTTIMEOUT[/-][+]sometimes transfers timeout earlier than CURLOPT_CONNECTTIMEOUT[/+]bagder commentedon Feb 22, 2019
I'm afraid the amount of PHP code plus PHP binding layering here is a bit too thick for me to tell exactly what's going on or where exactly the issue lies. We need this example converted to plain C so that we can check exactly what's going on and why. It seems odd that this issue should suddenly start to appear after a certain amount of parallel transfers. Is it out of sockets somehow? If you limit this program to only test for 100 proxies, does then always work correctly?
accountantM commentedon Feb 22, 2019
Thank you very much Daniel for your replay. I did more tests
test2_100proxies_bugAppear
test3_100proxies_bugAppear
test4_407proxies_NoBug
/test5_407proxies_BugAppears
test6_30proxies_NoBug
I did about 100 tests and the bug appeared in about 20 tests of them (I did the tests on both a production server Linux centos, and laptop Linux fedora)
regarding the sockets question, I don't know if they out of sockets or not, I just use the php curl_* functions, and I don't know what are they doing behind the scenes.
I also noted something. The reports that don't have the issue are having a normal total_time from the very FIRST request (e.g 1.999564 seconds while the CURLOPT_CONNECTTIMEOUT is 2 seconds ), and the reports with the issue are having late total_time from the very FIRST request (e.g 3.020935 seconds while the CURLOPT_CONNECTTIMEOUT is 2 seconds)
I don't know C but I will download a C compiler and an IDE and will try my best to write this example in plain C.
Thank you very much Daniel for your effort and time in cURL which is almost the only reliable solution for PHP developers for making parallel http requests
bagder commentedon Feb 23, 2019
I wrote a C adaptation of what I believe is the same spirit as the original PHP code. See issue-3602.c.
It does however not seem to reproduce the problem here when I use it with curl from current git master.
accountantM commentedon Feb 23, 2019
Thank you for making this C code, I will run it on the same 2 machines I run the php code on, and will tell you the results, just give me some time
accountantM commentedon Feb 24, 2019
It appeared again ! :(
I run your code one time then I added these 3 lines to your code
and run another time
I tested on my laptop a fedora with curl 7.59.0 and the same issue happened in the 2 tests, I will try to run the same code on my cantos production server curl 7.64.0, it has much more resources(CPU and internet connection ) than my laptop and tell you the results, if you want me to record my screen while doing the tests, just tell me. This is the output in the terminal
The Second Test result
The First Test Result
accountantM commentedon Feb 24, 2019
I tried more 10 times and it appeared in 3 times of them (on Linux Fedora curl 7.59)
accountantM commentedon Feb 24, 2019
I suppressed the output using the code in this example and allowed up to 30 seconds for connect and total 40 seconds for time out
and it appeared also several times (notice the requests with "Connection time-out" and small total time)
bagder commentedon Feb 24, 2019
Thanks, I'll dig deeper.
bagder commentedon Feb 24, 2019
I updated my example code. It now measures the elapsed time by itself as well and then outputs both the time libcurl tells for
TOTAL_TIMEas well as the private timer. It shows that the time reported by libcurl seems to be grossly wrong in these cases and the actual time that it actually took is pretty much what we set the timeout limit to. Example output from my tests would look like this:Here libcurl says it took 39000 microseconds for each of these transfers to fail, but the application itself measured them to 3036000 microseconds. 39 vs 3036 milliseconds.
I haven't yet figured out why this happens...
bagder commentedon Feb 24, 2019
Ok, I figured it out. The upside is that this is only
CURLINFO_TOTAL_TIMEreporting the wrong time, it isn't actually returning prematurely. PR coming right up.multi: call multi_done on connect timeouts
accountantM commentedon Feb 26, 2019
I'm sorry I was too late to respond to you, as I was doing some tests and making some reports for you.
First of all thank you very much Daniel for fixing this issue in this remarkable short time.
The Problem here - and it is my fault not to describe it better with a focus on the main issue - is that it is not the small
total_timeproblem that made me report this to you, it is why these requests are timing out? they shouldn't!I don't know much about cURL or C language but in the past 14 days I have used cURL multi interface for testing public proxies hundreds of thousands of times as part of my current project (a PHP application for gathering and testing the free/public proxies on the internet), and I'm telling you there is something unusual happening here in the multi feature and timing out that I don't understand.
these 400 proxies in the php and C code should have about 40-80 working proxies(and by working I mean I can connect to them in less than 30 seconds, they don't refuse or timeout) you can prove this by testing each one of them alone without the multi interface. but if I tested them with curl multi interface, SOMETIMES , I get toooooooooo many timeout errors 28 by curl, almost 3-7 proxies out of the 407 connects without errors.
HOW TO REPRODUCE:
1- get a list of free public proxies (like the ones in the C code or from websites like free-proxy-list.net) around 400 proxies
2- test each of them alone(without the multi interface) so you can estimate how many proxies of them is actually working (cURL errno 0) and how many timesout (errno 28) (I expect about 40 out of 400 will connect with no errors) (
CONNECT_TIMEOUT= 30)3- Now test them using the multi_interface and check how many timeout error (errno 28) happens and how many works (errno 0) (
CONNECT_TIMEOUT= 30)4-repeat number 3 until you suddenly find almost all of them timesout!! (here is the problem, why they are timing out sometimes ?)
I run an automatic test by a cron job every 2 minutes for the past ~24 hours by the a C code and PHP code from my laptop and a production server and saved the results in this file, you can see that some tests reproduces the problem (too many 28 errors almost all the proxies, while the normal results should be about 80 proxies are working without errors)
I'm sorry I couldn't focus on the main issue here as the
total_timeproblem got all the attentionThis is my first time I use github and I don't know if this should be reported as new issue or not and if you will get notified about this post or not as it is closed
11 remaining items
accountantM commentedon Mar 29, 2019
I'm sorry I don't clearly understand what you mean, do you mean that this C threads program can help in the issue ?
JL2210 commentedon Mar 30, 2019
@accountantM It may not be my place to make a comment here, but I think that maybe you shouldn't hard press this issue. One aspect of it has been fixed, and the other is only reproducible very rarely. Possibly try changing your network connection and try again, but I certainly doubt this is work the 100 bounty you've given it on Stack Overflow. Possibly just close the Stack Overflow question and wait every so often before you go knocking on the doors of ~400 proxy sites?
One more thing: Check your bandwidth. I know a few issues that I've had with curl simply go away at times when there is low bandwidth in my area.
Hope you can figure out your issue 👍.
accountantM commentedon Mar 30, 2019
WOW, this is very enough for me to set 500 bounty on the problem not only 100, I can't rely on curl_multi since it has a problem that shows up rarely!, for me I will just continue my project using curl on threads, and leave this problem for the time, may be one day Daniel give it another deeper look.
No, I should, curl is great and that is why I spent too much time and effort on this problem, because I want to see curl perfect and I'm very sure there is something wrong in curl_multi and proxies and this area.
thanks :)
JL2210 commentedon Mar 30, 2019
Doesn't Curl have a threaded resolver? Is that what you mean by "curl on threads"?
Why do you need to test these proxies, anyway? Just find one that works when the problem doesn't show itself, and use that.Read the SO question more carefully. Why not test every one individually if that works?If Curl's multi-threaded resolver works for you, then just use that.
I'll try and reproduce and get back to you. What network connection are you on?
accountantM commentedon Mar 30, 2019
Ummm, I don't know!
No, what I meant by "curl on threads" is the C program that I wrote that uses curl on threads for testing the same set of proxies 407.
the curl_multi program sometimes shows the issue
the curl on threads program never shows the issue
The 2 programs are here, and the tests results are here
I'm working on a project which I want to gather the public/free proxies from the internet, save them in my database, test them , rank them (you can find plenty of them if you googled free-proxy-list )
because testing and using thousands of public proxies synchronously will take ages, as you know they are very slow, most of them will take 60 seconds tell it responds or timeout, and also because after this project I'm going to make a crawler that will use these proxies, and for the same reason a crawler MUST do it's requests in parallel to save time.
Yes, I will definitely do that since curl_multi interface has this issue
Thank you very much for that ❤️
I tested from my production server from a network in the U.S and from my laptop from a network in Egypt. The production server offcurse has more resources (CPU/network) than my development laptop, however curl_multi still shows the problem sometimes on both machines
JL2210 commentedon Mar 30, 2019
Enable the threaded resolver by passing
--enable-threaded-resolverto Curl'sconfigurescript.I have three instances of that program running in the background with all their output redirected to logs, I'll get back to you on how it goes.
JL2210 commentedon Mar 30, 2019
Wow! All three consistently behave the same way! Here's the three logs:
log3.txt
log2.txt
log1.txt
I'm on a mobile hotspot in Indiana, if that helps.
JL2210 commentedon Mar 30, 2019
Strace output about where all the 28 errors are:
@bagder I'm getting this behavior consistently, and I can provide more debug info. If you don't mind, could you reopen this?
I'm building Wireshark and latest Curl right now, I'll give you the output as soon as possible.
accountantM commentedon Mar 30, 2019
I see you are getting 0 successful connections and about ~270 timeouts, If you tried the threads program you will see the normal behaviour "about 50 proxies will work and about only 200 timeouts". Thank you veeeeeeeery much for reproducing this
JL2210 commentedon Mar 30, 2019
Curl version
7.64.1configured withTrying to build right now.
JL2210 commentedon Mar 30, 2019
I got hung up on building Wireshark. Is there any alternative that doesn't require Qt?
JL2210 commentedon Mar 30, 2019
Latest Curl binary exhibits the same problem.
accountantM commentedon Mar 31, 2019
I downloaded Wireshark and used it to capture the traffic while each of the 2 C programs was running, I also filtered the traffic to the proxies list used by the 2 C programs, and saved the files on GitHub.
the curl-threads program (the expected behavior)
63 successful connections and 158 connections timeout
error 28out of 407 proxies.the curl_multi program (the unexpected behavior)
0 successful connections and 272 connections timeout
error 28out of 407 proxies.(You can open the .pcapng files using wireshark and see the recorded traffic on my computer while both expected/unexpected behavior (I filtered the traffic to the 407 proxies IPs and left Wireshark open for a little while after the 30 seconds of curl limit because I saw some packets still showing up, I don't know Wireshark and this level of networking, but i thought this could be useful))
@bagder I noticed in Wireshark that some proxy servers do reply with some packets, however curl_muti reported 0 successful connections, I don't know if this is normal replying package or not.
Hey Daniel, I'm sorry if I looked rude, It is not what I meant !
accountantM commentedon Mar 31, 2019
I don't know if this info is important or not, check the conversation with the proxy
92.244.36.66in the normal behaviour (63 successful connections) , and the up normal behaviour (0 successful connections ) , I copied them from the wireshark files I attached.(Please note the second column, which is the time in seconds since the beginning of capture)
This is a normal conversation(I think) done by the threads program
And this is a not normal conversation(I think) done by the curl_multi program
Did you see that? the time of the last packet sent from my computer (30.055897223) while the time of the packet before it is (4.905076421)! Why my computer took all these seconds to send the packet in the curl_multi program while it didn't in the curl threads program ? I don't know if this is normal or not, it might be useful or not.