sometimes transfers timeout earlier than CURLOPT_CONNECTTIMEOUT #3602
Labels
Comments
bagder
changed the title
|
I'm afraid the amount of PHP code plus PHP binding layering here is a bit too thick for me to tell exactly what's going on or where exactly the issue lies. We need this example converted to plain C so that we can check exactly what's going on and why. It seems odd that this issue should suddenly start to appear after a certain amount of parallel transfers. Is it out of sockets somehow? If you limit this program to only test for 100 proxies, does then always work correctly? |
|
Thank you very much Daniel for your replay. I did more tests I did about 100 tests and the bug appeared in about 20 tests of them (I did the tests on both a production server Linux centos, and laptop Linux fedora) regarding the sockets question, I don't know if they out of sockets or not, I just use the php curl_* functions, and I don't know what are they doing behind the scenes. I also noted something. The reports that don't have the issue are having a normal total_time from the very FIRST request (e.g 1.999564 seconds while the CURLOPT_CONNECTTIMEOUT is 2 seconds ), and the reports with the issue are having late total_time from the very FIRST request (e.g 3.020935 seconds while the CURLOPT_CONNECTTIMEOUT is 2 seconds) I don't know C but I will download a C compiler and an IDE and will try my best to write this example in plain C. |
|
I wrote a C adaptation of what I believe is the same spirit as the original PHP code. See issue-3602.c. It does however not seem to reproduce the problem here when I use it with curl from current git master. |
|
Thank you for making this C code, I will run it on the same 2 machines I run the php code on, and will tell you the results, just give me some time |
|
It appeared again ! :( and run another time I tested on my laptop a fedora with curl 7.59.0 and the same issue happened in the 2 tests, I will try to run the same code on my cantos production server curl 7.64.0, it has much more resources(CPU and internet connection ) than my laptop and tell you the results, if you want me to record my screen while doing the tests, just tell me. This is the output in the terminal The Second Test resultThe First Test Result |
|
I tried more 10 times and it appeared in 3 times of them (on Linux Fedora curl 7.59) |
|
I suppressed the output using the code in this example and allowed up to 30 seconds for connect and total 40 seconds for time out and it appeared also several times (notice the requests with "Connection time-out" and small total time) |
|
Thanks, I'll dig deeper. |
|
I updated my example code. It now measures the elapsed time by itself as well and then outputs both the time libcurl tells for Here libcurl says it took 39000 microseconds for each of these transfers to fail, but the application itself measured them to 3036000 microseconds. 39 vs 3036 milliseconds. I haven't yet figured out why this happens... |
|
Ok, I figured it out. The upside is that this is only |
bagder
added a commit
that referenced
this issue
Feb 24, 2019
Failing to do so would make the CURLINFO_TOTAL_TIME timeout to not get updated correctly and could end up getting reported to the users to be completely wrong (way too small). Reported-by: accountantM on github Fixes #3602
|
I'm sorry I was too late to respond to you, as I was doing some tests and making some reports for you. First of all thank you very much Daniel for fixing this issue in this remarkable short time. The Problem here - and it is my fault not to describe it better with a focus on the main issue - is that it is not the small I don't know much about cURL or C language but in the past 14 days I have used cURL multi interface for testing public proxies hundreds of thousands of times as part of my current project (a PHP application for gathering and testing the free/public proxies on the internet), and I'm telling you there is something unusual happening here in the multi feature and timing out that I don't understand. these 400 proxies in the php and C code should have about 40-80 working proxies(and by working I mean I can connect to them in less than 30 seconds, they don't refuse or timeout) you can prove this by testing each one of them alone without the multi interface. but if I tested them with curl multi interface, SOMETIMES , I get toooooooooo many timeout errors 28 by curl, almost 3-7 proxies out of the 407 connects without errors. HOW TO REPRODUCE: 1- get a list of free public proxies (like the ones in the C code or from websites like free-proxy-list.net) around 400 proxies 2- test each of them alone(without the multi interface) so you can estimate how many proxies of them is actually working (cURL errno 0) and how many timesout (errno 28) (I expect about 40 out of 400 will connect with no errors) ( 3- Now test them using the multi_interface and check how many timeout error (errno 28) happens and how many works (errno 0) ( 4-repeat number 3 until you suddenly find almost all of them timesout!! (here is the problem, why they are timing out sometimes ?) I run an automatic test by a cron job every 2 minutes for the past ~24 hours by the a C code and PHP code from my laptop and a production server and saved the results in this file, you can see that some tests reproduces the problem (too many 28 errors almost all the proxies, while the normal results should be about 80 proxies are working without errors) I'm sorry I couldn't focus on the main issue here as the This is my first time I use github and I don't know if this should be reported as new issue or not and if you will get notified about this post or not as it is closed |
|
You're just randomly trying remote sites if they work they way you want them to, and you do that at different times and you draw conclusions and seem to think that their behavior should be predictable and remain stable. I don't think you can do that since you have no idea about their logic, load, DOS-preventions or other decisions they make or do that changes connection times and success rate over time. I've found nothing in curl that indicates that there's a problem in its connection logic or the timeout handling of those connect attempts. I believe you're seeing and experience genuine connection timeouts because your local TCP stack's connection attempt is not completed within the given time restrictions. I'm not suggesting that there can't be any errors in curl in this area, but the method you're using is next to useless in proving that there is. If you can reproduce such timeouts against a server we know for sure is up and accepting and for which we can inspect the traffic and logs, then I'm certainly willing and very keen in digging further on this topic. But as long as you're connecting to random sites on the Internet that you have no idea about their inner workings, this is a very weak case. |
|
Of course, you can also monitor your TCP connections with a tool like Wireshark to see what's actually happening on the TCP level. Do they actually ever connect without curl noticing within the timeout period? If so, that's a curl bug. If not, then curl is right. Doing that on a massive amount of parallel connections will of course be tricky and hard. |
I don't have such amount of proxies that is under my control, but we know there are about 70~90 are working out of the same 400 proxies because if I tried each of them alone without the curl_multi I get the same rate, and curl_multi itself most of the times make the same rate, however SOMETIMES curl_multi timeout ALMOST ALL OF THEM (they are different servers from different networks in different countries, it can't be something wrong, like protection or load suddenly happens to all of them) Just take a deeper look on column 5(number of connections without errors) and column 7(number of connections with timeout error 28) out of the 407 proxies, and you will notice that there are about 80 proxies are working but sometimes curl_multi says almost all of them timeout!. For example here is the tests from 93 to 100 Is this normal ? curl_multi most of the times report 97/90 (noeror/error28) rate(the same rate I always get if I tried each of them alone/synchronously without curl_multi) and sometimes curl_multi report(4/229)! I understand this is very hard to debug since we don't control the proxies servers, I'm just reporting to you what I found. I'm also going to write a multi threading code that uses curl for 1 request per thread without the curl_multi interface and do the same tests to be sure about the works/timeouts rate if I didn't use curl_multi |
the timeouts are happening to the connections to the proxies ( |
I can do this easily on my production server if you want me to, I can prepare it to be the final destination server as a replacement to www.example.com that all the proxies will connect to, and log every request came from these proxies the way you want (Actually this is the heart of my public proxies testing application) For example I can replace example.com with |
I done the tests from 3 different sources (the C code on my laptop, the PHP code on my laptop, and a PHP code on my server, my laptop is from a network in Egypt and the server is from a network in the US). Notice the time of the tests in the file and you will see that at the same time the C code reproduced the problem, the PHP code works fine, or the other way, or the laptop reproduce and in the same time the server works fine, this is an indicator that the problem is in my program not the proxies. |
|
Again, I can't rule out that there is a problem somewhere in curl. But without reliable, reproducible behavior I don't know how to determine when a timeout is "real" or not by debugging curl. Do the TCP handshakes complete within the timeout period or not. Is it curl that doesn't notice the connection or is the connection not happening (fast enough). Those are the questions we want answered. That would probably require that you analyze the network traffic. Continuing to post times or aggregated data won't help us to get those questions answered. They're just observations that might show a curl issue, but they might also just be problems elsewhere. |
|
Another approach, which could be better but probably also hard to pull off, would be to somehow setup proxies with fixed and controlled behavior that perhaps would have various degrees of slowness in the TCP handshakes, and then we could run curl against such a setup and see how it behaves when doing this using many parallel connections down to only one. |
I'm sorry Daniel, I'm not such a deep networks developer, I'm a PHP/MySQL developer with only 4 years of experience in developing ERP systems (my area of experience is in sales, purchases, inventory, accounting and stuff like that) I merely can open wireshark and watch the traffic at such low level for fun(I don't understand 95% of the things I see in wireshark) I even took about 8 hours just to remember how to compile a simple C file. Anyway, I guess this is the end of this issue for me, thank you very much for your interest and for fixing the |
|
As you know I can't do more regarding this issue, but I wrote a C program that uses curl with threading and tested the same proxies, and the threads program never shows the timeout issue. I asked for more help from the community and I hope this issue gets your attention |
Apparently you could... |
I'm sorry I don't clearly understand what you mean, do you mean that this C threads program can help in the issue ? |
|
@accountantM It may not be my place to make a comment here, but I think that maybe you shouldn't hard press this issue. One aspect of it has been fixed, and the other is only reproducible very rarely. Possibly try changing your network connection and try again, but I certainly doubt this is work the 100 bounty you've given it on Stack Overflow. Possibly just close the Stack Overflow question and wait every so often before you go knocking on the doors of ~400 proxy sites? Hope you can figure out your issue |
WOW, this is very enough for me to set 500 bounty on the problem not only 100, I can't rely on curl_multi since it has a problem that shows up rarely!, for me I will just continue my project using curl on threads, and leave this problem for the time, may be one day Daniel give it another deeper look.
No, I should, curl is great and that is why I spent too much time and effort on this problem, because I want to see curl perfect and I'm very sure there is something wrong in curl_multi and proxies and this area.
thanks :) |
|
Doesn't Curl have a threaded resolver? Is that what you mean by "curl on threads"? |
Ummm, I don't know!
No, what I meant by "curl on threads" is the C program that I wrote that uses curl on threads for testing the same set of proxies 407. the curl_multi program sometimes shows the issue The 2 programs are here, and the tests results are here
I'm working on a project which I want to gather the public/free proxies from the internet, save them in my database, test them , rank them (you can find plenty of them if you googled free-proxy-list )
because testing and using thousands of public proxies synchronously will take ages, as you know they are very slow, most of them will take 60 seconds tell it responds or timeout, and also because after this project I'm going to make a crawler that will use these proxies, and for the same reason a crawler MUST do it's requests in parallel to save time.
Yes, I will definitely do that since curl_multi interface has this issue
Thank you very much for that
I tested from my production server from a network in the U.S and from my laptop from a network in Egypt. The production server offcurse has more resources (CPU/network) than my development laptop, however curl_multi still shows the problem sometimes on both machines |
|
Enable the threaded resolver by passing |
|
Strace output about where all the 28 errors are: @bagder I'm getting this behavior consistently, and I can provide more debug info. If you don't mind, could you reopen this? I'm building Wireshark and latest Curl right now, I'll give you the output as soon as possible. |
|
I see you are getting 0 successful connections and about ~270 timeouts, If you tried the threads program you will see the normal behaviour "about 50 proxies will work and about only 200 timeouts". Thank you veeeeeeeery much for reproducing this |
|
Curl version Trying to build right now. |
|
I got hung up on building Wireshark. Is there any alternative that doesn't require Qt? |
|
Latest Curl binary exhibits the same problem. |
|
I downloaded Wireshark and used it to capture the traffic while each of the 2 C programs was running, I also filtered the traffic to the proxies list used by the 2 C programs, and saved the files on GitHub. the curl-threads program (the expected behavior) 63 successful connections and 158 connections timeout
the curl_multi program (the unexpected behavior) 0 successful connections and 272 connections timeout
(You can open the .pcapng files using wireshark and see the recorded traffic on my computer while both expected/unexpected behavior (I filtered the traffic to the 407 proxies IPs and left Wireshark open for a little while after the 30 seconds of curl limit because I saw some packets still showing up, I don't know Wireshark and this level of networking, but i thought this could be useful)) @bagder I noticed in Wireshark that some proxy servers do reply with some packets, however curl_muti reported 0 successful connections, I don't know if this is normal replying package or not. Hey Daniel, I'm sorry if I looked rude, It is not what I meant ! |
|
I don't know if this info is important or not, check the conversation with the proxy (Please note the second column, which is the time in seconds since the beginning of capture) This is a normal conversation(I think) done by the threads program And this is a not normal conversation(I think) done by the curl_multi program Did you see that? the time of the last packet sent from my computer (30.055897223) while the time of the packet before it is (4.905076421)! Why my computer took all these seconds to send the packet in the curl_multi program while it didn't in the curl threads program ? I don't know if this is normal or not, it might be useful or not. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I did this
used curl_multi interface in php to test about 400 proxies at the same time with a CURLOPT_CONNECTTIMEOUT set 30, however sometimes the later requests (about 150 from 400) all outputs "Connection time-out" and the total_time is very small (about 0.002 s)!
this is a question on stackoverflow.com
and this is a php class that tests this behaviour CurlMultiTimeoutBugTester
This is a sample output from the tester, please notice the later requests
I expected the following
every request takes his chance and tries to connect for 30 seconds then outputs "Connection timed out after 30005 milliseconds" and total_time is 30 seconds or succeed
curl/libcurl version
7.59.0 and 7.64.0
operating system
Linux CentOS and Linux fedora
The text was updated successfully, but these errors were encountered: