Upgrading curl from 7.86.0 to 8.0.1 causing a lot of logs shown "Connection #xxx is still name resolving, can't reuse"

[yushicheng7788]

I did this

I ran the example code https://curl.se/libcurl/c/10-at-a-time.html and replaced the urls array in the code with urls from the same host. There are a lot of logs showing "connection #xxx is still name resolving, can't reuse". After a series of investigations, I found that this commit #10456 caused it. Is this a normal phenomenon normal? It seems that this change will cause conn->bits.close to be false, so the if (check->connect_only || check->bits.close) of the ConnectionExists function is judged to be false, causing the function to continue to execute, and a large number of logs appear " Connection #xxx is still name resolving, can't reuse". If the change is rolled back, there will be no log indicating that the connection cannot be reused.

I expected the following

I would like to know whether this phenomenon is normal or if there is an error.

curl/libcurl version

8.0.1

operating system

Mac / Windows

[yushicheng7788]

@icing Could you explain this problem? Thank you!

[icing]

I see no verbose output from 10-at-a-time. Have you modified it further than changing the URLs? I am having trouble reproducing this.

The message you mention is only emitted when an async resolver is configured, right? My question is then, will existing connections be correctly reused or is a new connection opened for every URL as long as the resolver does not return?

[yushicheng7788]

Thank you for your reply. Blow is test example source code.

#include <errno.h>
#include <stdlib.h>
#include <string>
#include <iostream>
#ifndef WIN32
#include <unistd.h>
#endif
#include <curl/curl.h>

static const char* urls[] = {
	"https://www.man7.org/linux/man-pages/man2/socket.2.html",
	"https://www.man7.org/linux/man-pages/man2/accept.2.html",
	"https://www.man7.org/linux/man-pages/man2/bind.2.html",
	"https://www.man7.org/linux/man-pages/man2/getsockname.2.html",
	"https://www.man7.org/linux/man-pages/man3/sd_is_fifo.3.html",
	"https://www.man7.org/linux/man-pages/man2/getpeername.2.html",
	"https://www.man7.org/linux/man-pages/man7/ip.7.html",
	"https://www.man7.org/linux/man-pages/man7/udp.7.html",
	"https://www.man7.org/linux/man-pages/man2/sendmsg.2.html",
	"https://www.man7.org/linux/man-pages/man2/write.2.html",
	"https://www.man7.org/linux/man-pages/man2/lseek.2.html",
};

#define MAX_PARALLEL 10L /* number of simultaneous transfers */
#define NUM_URLS sizeof(urls)/sizeof(char *)

#define MAX_LIBCURL_LOG_DATA_SIZE	(1024*4)
int my_curl_debug_callback(CURL* handle,      /* the handle/transfer this concerns */
	curl_infotype type, /* what kind of data */
	char* data,        /* points to the data */
	size_t size,       /* size of the data pointed to */
	void* userptr)    /* whatever the user please */
{
	if (size <= MAX_LIBCURL_LOG_DATA_SIZE)
	{
		std::string str(data, size);
		std::cout << str << std::endl;
	}
	return 0;
}

static size_t write_cb(char* data, size_t n, size_t l, void* userp)
{
	/* take care of the data here, ignored in this example */
	(void)data;
	(void)userp;
	return n * l;
}

static void add_transfer(CURLM* cm, int i, int* left)
{
	CURL* eh = curl_easy_init();
	curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
	curl_easy_setopt(eh, CURLOPT_URL, urls[i]);
	curl_easy_setopt(eh, CURLOPT_PRIVATE, urls[i]);
	curl_easy_setopt(eh, CURLOPT_SSL_VERIFYPEER, 0L);
	curl_easy_setopt(eh, CURLOPT_VERBOSE, 1L);
	curl_easy_setopt(eh, CURLOPT_DEBUGFUNCTION, my_curl_debug_callback);
	curl_multi_add_handle(cm, eh);
	(*left)++;
}

int main(void)
{
	CURLM* cm;
	CURLMsg* msg;
	unsigned int transfers = 0;
	int msgs_left = -1;
	int left = 0;

	curl_global_init(CURL_GLOBAL_ALL);
	cm = curl_multi_init();

	/* Limit the amount of simultaneous connections curl should allow: */
	curl_multi_setopt(cm, CURLMOPT_MAXCONNECTS, (long)MAX_PARALLEL);

	for (transfers = 0; transfers < MAX_PARALLEL && transfers < NUM_URLS;
		transfers++)
		add_transfer(cm, transfers, &left);

	do {
		int still_alive = 1;
		curl_multi_perform(cm, &still_alive);

		while ((msg = curl_multi_info_read(cm, &msgs_left))) {
			if (msg->msg == CURLMSG_DONE) {
				char* url;
				CURL* e = msg->easy_handle;
				curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE, &url);
				fprintf(stderr, "R: %d - %s <%s>\n",
					msg->data.result, curl_easy_strerror(msg->data.result), url);
				curl_multi_remove_handle(cm, e);
				curl_easy_cleanup(e);
				left--;
			}
			else {
				fprintf(stderr, "E: CURLMsg (%d)\n", msg->msg);
			}
			if (transfers < NUM_URLS)
				add_transfer(cm, transfers++, &left);
		}
		if (left)
			curl_multi_wait(cm, NULL, 0, 1000, NULL);

	} while (left);

	curl_multi_cleanup(cm);
	curl_global_cleanup();

	return EXIT_SUCCESS;
}

We configured async resolver. A completed conn can be reused. But if you add multiple easy handles belonging to the same host to a multi handle, and then execute curl_multi_perform(), a large number of "Connection #xxx is still name resolving, can't reuse" will appear.

[yushicheng7788]

Maybe this is a normal behavior, but would like to get your confirmation.

[icing]

I think this verbose output should not happen in your example. We should change this. The change you referenced was necessary if the application uses CURLOPT_PIPEWAIT, e.g. really wants to use HTTP/2's ability to process several transfers on the same connection.

But you are not usingCURLOPT_PIPEWAIT, meaning you are ok with every transfer opening a new connection. In this context, the verbose output you see is not interesting. Not harmful either, but...

Let's see what @bagder has to say to this when he is back.

[yushicheng7788]

Thank you for your explanation.