Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upDNS_CACHE_TIMEOUT == 0 but address might still be used for second transfer! #2169
Comments
|
To reduce some of the confusion and complexity:
|
|
I wrote up the little example below in an attempt to reproduce, but I can't see it fail. Can you figure out what needs to be changed to reproduce a problem? #include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
struct curl_slist *dns;
dns = curl_slist_append(NULL, "example.com:80:127.0.0.102");
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_HEADER, 1L);
curl_easy_setopt(curl, CURLOPT_RESOLVE, dns);
curl_easy_setopt(curl, CURLOPT_DNS_CACHE_TIMEOUT, 0L);
res = curl_easy_perform(curl);
curl_slist_free_all(dns);
/* remove the old mapping, add a new */
dns = curl_slist_append(NULL, "-example.com:80");
dns = curl_slist_append(dns, "example.com:80:127.0.0.99");
curl_easy_setopt(curl, CURLOPT_RESOLVE, dns);
curl_easy_setopt(curl, CURLOPT_FRESH_CONNECT, 1L);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_slist_free_all(dns);
}
return 0;
} |
|
Your example matches my expectations, and I figured DNS cache was separate from connection reuse. The question I still have is what is the point of FORBID_REUSE if 1) I set it before any transfers and 2) second transfer uses the first transfer's connection. |
|
FORBID_REUSE should do exactly what it says. The second transfer should not reuse the connection from the first transfer. Can you give an example in C? edit: after reading your example it occurs to me maybe you misunderstand. It looks like you are doing this: res = curl_easy_perform(curl); // this transfer's connection will be reused
curl_easy_setopt(curl, CURLOPT_FORBID_REUSE, 1L);
res = curl_easy_perform(curl); // this transfer's connection won't be reusedbut really you want this: curl_easy_setopt(curl, CURLOPT_FORBID_REUSE, 1L);
res = curl_easy_perform(curl); // this transfer's connection won't be reused
res = curl_easy_perform(curl); // this transfer's connection won't be reused |
|
@jay The last example I posted issues FORBID_REUSE before the first transfer. |
|
Ok. I cannot reproduce that here. Can you give us an example in C? |
|
Investigating, looks like the C code works as I expect and as you stated here. |
|
My previous example was a user error, but I think this is legit. The following code connects to the real ip the second time. #include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
struct curl_slist *dns;
//dns = curl_slist_append(NULL, "example.com:80:127.0.0.102");
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_HEADER, 1L);
//curl_easy_setopt(curl, CURLOPT_RESOLVE, dns);
curl_easy_setopt(curl, CURLOPT_DNS_CACHE_TIMEOUT, 0L);
res = curl_easy_perform(curl);
//curl_slist_free_all(dns);
/* remove the old mapping, add a new */
//dns = curl_slist_append(NULL, "-example.com:80");
dns = curl_slist_append(NULL, "example.com:80:127.0.0.99");
curl_easy_setopt(curl, CURLOPT_RESOLVE, dns);
curl_easy_setopt(curl, CURLOPT_FRESH_CONNECT, 1L);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_slist_free_all(dns);
}
return 0;
} |
|
That is different from FORBID_REUSE. Your example does not reuse the connection, however it appears to be skipping the 127.0.0.99 entry or zapping the wrong one maybe, which is why you get two connections to the same ip. That looks wrong I'll investigate |
|
Yep thanks. |
|
libcurl always puts entries into the dns cache, even when the timeout is zero, since the logic everywhere assumes it. And it remains in there as long as it is in use by the current transfer. The problem here is that we don't prune the cache immediately when the transfer is done but only when it disconnects, so in this case the first host name resolve remains in the cache when the second is tried and the This little patch that prunes the dns cache in diff --git a/lib/multi.c b/lib/multi.c
index 1a4618eb2..e35b8fa19 100644
--- a/lib/multi.c
+++ b/lib/multi.c
@@ -550,10 +550,11 @@ static CURLcode multi_done(struct connectdata **connp,
if(conn->dns_entry) {
Curl_resolv_unlock(data, conn->dns_entry); /* done with this */
conn->dns_entry = NULL;
}
+ Curl_hostcache_prune(data);
/* if the transfer was completed in a paused state there can be buffered
data left to free */
for(i = 0; i < data->state.tempcount; i++) {
free(data->state.tempwrite[i].buf); |
bagder
changed the title
|
|
Consider this:
If I remove fake dns entry a request is made to the real ip:
But now if I set another fake ip, curl makes a connection to the real ip:
OK, maybe I need to also set FORBID_REUSE. Let's try that.
Let's try this all again:
I think something in the DNS cache/connection reuse does not interact properly with RESOLVE option.
This came out of me investigating pycurl/pycurl#443 which, strictly speaking, appears to be a user error as the user did not disable DNS cache and connection reuse.