Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libcurl: 6765e6d breaks dns resolution in my setup with c-ares #4893

Closed
ernstp opened this issue Feb 7, 2020 · 18 comments
Closed

libcurl: 6765e6d breaks dns resolution in my setup with c-ares #4893

ernstp opened this issue Feb 7, 2020 · 18 comments
Labels

Comments

@ernstp
Copy link

@ernstp ernstp commented Feb 7, 2020

I did this

Curl compiled for arm with yocto like this:
Host setup: arm-fslc-linux-gnueabi
Install prefix: /usr
Compiler: arm-fslc-linux-gnueabi-gcc -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a7 -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/work/build/tmp/work/cortexa7t2hf-neon-fslc-linux-gnueabi/curl/7.64.1-r0/recipe-sysroot
CFLAGS: -O2 -pipe -feliminate-unused-debug-types -fmacro-prefix-map=/work/build/tmp/work/cortexa7t2hf-neon-fslc-linux-gnueabi/curl/7.64.1-r0=/usr/src/debug/curl/7.64.1-r0 -fdebug-prefix-map=/work/build/tmp/work/cortexa7t2hf-neon-fslc-linux-gnueabi/curl/7.64.1-r0=/usr/src/debug/curl/7.64.1-r0 -fdebug-prefix-map=/work/build/tmp/work/cortexa7t2hf-neon-fslc-linux-gnueabi/curl/7.64.1-r0/recipe-sysroot= -fdebug-prefix-map=/work/build/tmp/work/cortexa7t2hf-neon-fslc-linux-gnueabi/curl/7.64.1-r0/recipe-sysroot-native= -Werror-implicit-function-declaration -Wno-system-headers
CPPFLAGS:
LDFLAGS: -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now
LIBS: -lcares

curl version: 7.64.1
SSL: no (--with-{ssl,gnutls,nss,polarssl,mbedtls,cyassl,schannel,secure-transport,mesalink,amissl} )
SSH: no (--with-libssh2)
zlib: no (--with-zlib)
brotli: no (--with-brotli)
GSS-API: no (--with-gssapi)
TLS-SRP: no (--enable-tls-srp)
resolver: c-ares
IPv6: no (--enable-ipv6)
Unix sockets: enabled
IDN: no (--with-{libidn2,winidn})
Build libcurl: Shared=yes, Static=no
Built-in manual: no (--enable-manual)
--libcurl option: no
Verbose errors: enabled (--disable-verbose)
Code coverage: disabled
SSPI: no (--enable-sspi)
ca cert bundle: no
ca cert path:
ca fallback:
LDAP: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS: no (--enable-ldaps)
RTSP: no (--enable-rtsp)
RTMP: no (--with-librtmp)
Metalink: no (--with-libmetalink)
PSL: no (--with-libpsl)
Alt-svc: no (--enable-alt-svc)
HTTP2: disabled (--with-nghttp2)
Protocols: FILE FTP HTTP
Features: UnixSockets AsynchDNS

I expected the following

I do something like this:
curl_easy_setopt(curl_master, CURLOPT_DNS_SERVERS, "1.2.3.4,1.2.3.5");
new_handle = curl_easy_duphandle(curl_master);
setup request on new_handle
curl_easy_perform(new_handle);

  • Could not resolve: foo.bar.stuff (Domain name not found)
  • Closing connection 0
    curl_easy_perform failed: Couldn't resolve host name (6)

curl/libcurl version

I notice this when upgrading from libcurl 7.62 to 7.64. I took a guess and reverted 6765e6d and that made it work again.
My setup also doesn't work with 7.68.

operating system

Linux Yocto 2.7 kernel 4.14 armv7l

@bagder bagder added the name lookup label Feb 7, 2020
@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 7, 2020

Can you be more specific? Can you deduct what causes the issue and produce a full demo source code that reproduces the issue for you when you run it? That way we would a good way to debug this. Simply reverting that commit is too crude and will only bring back the bug it fixed.

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

Sure, I'll try to make a small demo. Sharing what I've done so far and looking for possible tips or clues.

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

So
curl_easy_setopt(curl_master, CURLOPT_DNS_SERVERS, "1.2.3.4,1.2.3.5");
new_handle = curl_easy_duphandle(curl_master);
curl_easy_perform(new_handle);

...doesn't work anymore, but:

new_handle = curl_easy_duphandle(curl_master);
curl_easy_setopt(curl_master, CURLOPT_DNS_SERVERS, "1.2.3.4,1.2.3.5");
curl_easy_perform(new_handle);

...still works.

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

Ok this was actually pretty easy to write a testcase for.

This is the output I expect:

LD_LIBRARY_PATH=/home/ernsjo/curltest/install-762/lib/ ./test 
hello libcurl/7.62.0-DEV OpenSSL/1.0.2n zlib/1.2.11 c-ares/1.14.0 libidn2/2.0.4
* Could not resolve: really.not.a.server (Could not contact DNS servers)
* Closing connection 0

However with 7.63+ I get:

LD_LIBRARY_PATH=/home/ernsjo/curltest/install/lib/ ./test 
hello libcurl/7.69.0-DEV OpenSSL/1.0.2n zlib/1.2.11 c-ares/1.14.0 libidn2/2.0.4
* Could not resolve: really.not.a.server (Domain name not found)
* Closing connection 0

Here it's forgotten all about my special DNS server.

test.c.txt

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

Oh I didn't realize there actually is a DNS server on 9.9.9.9, but the port is wrong at least. :-)
Bugs in test.c aside, we want the "Could not contact DNS servers" print.

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 10, 2020

Clearly, options in the CURLOPT_DNS_* family don't survive a call to curl_easy_duphandle() so when you use the duplicate you get the c-ares default DNS settings. Are you saying this actually did the right thing back in 7.63.0 ?

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

In 7.62.0 it did the right thing yes. In 7.63.0 and higher it does not work.

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 10, 2020

Are you sure this is actually the same problem then? Because libcurl never duplicated the CURLOPT_DNS_* options when you called curl_easy_duphandle() which is quite easy to verify by reading the code.

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

Well you can see that test.c prints "Could not contact DNS servers" with 7.62.

Ok so you're saying that I'm using the api in a wrong way then? I haven't been able to clearly find this in the documentation (which is otherwise very thorough).

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 10, 2020

This code:

  curl_master = curl_easy_init();
  curl_easy_setopt(curl_master, CURLOPT_URL, "https://curl.haxx.se/");
  curl_easy_setopt(curl_master, CURLOPT_DNS_SERVERS, "1.2.3.4,1.2.3.5");
  new_handle = curl_easy_duphandle(curl_master);
  ret = curl_easy_perform(new_handle);

Will simply use the default name server c-ares figures out, not the one set in the master handle.

Thus a simpler code that does the same thing is:

  curl_master = curl_easy_init();
  curl_easy_setopt(curl_master, CURLOPT_URL, "https://curl.haxx.se/");
  ret = curl_easy_perform(curl_master);
@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 10, 2020

Will simply use the default name server c-ares figures out, not the one set in the master handle.

Not with 7.62, I clearly see that it uses the one from the master handle.

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 10, 2020

Ok, I read you. Will dig deeper and see what I'm missing...

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 10, 2020

I found the issue. You were of course right and I didn't dig deep enough until now.

In the commit that closed #3238, the code changed the the function Curl_resolver_duphandle() which previously used ares_dup() to duplicate the c-ares handle and then it would duplicate even the custom set DNS servers.

In the fix, the code changed that to instead do a fresh ares_init which of course then loses the custom DNS servers... The fix does this because apparently: "ares_dup does not allow passing a different sock_state_cb_data". Which of course is a c-ares bug (or missing feature perhaps) rather than anything else.

Now, for the best step to fix this I need to think a little...

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 17, 2020

I think this should be fixed by

  1. adding the necessary API to c-ares so that we can use ares_dup() in there
  2. until (1) is done, add a work-around in curl that stores the DNS server string in the easy handle and when duplicated, it can set them for the newly created cares channel.

What do you think @ernstp?

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 19, 2020

@bagder Sure, 2) sounds lightweight enough, and nice and generic.

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Feb 21, 2020

@ernstp do you feel up for working on this?

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Feb 28, 2020

@bagder Yeah it looks like I can give it a shot.

@ernstp

This comment has been minimized.

Copy link
Author

@ernstp ernstp commented Mar 2, 2020

@bagder Should (2) be ifdefed so it only exists when you compile with cares? always do if for all backends?

ernstp pushed a commit to ernstp/curl that referenced this issue Mar 2, 2020
With ares the dns parameters lives in ares_channel. Store them in the
curl handle so we can set them again if we do easy_duphandle.

Closes curl#4893

Signed-off-by: Ernst Sjöstrand <ernst.sjostrand@verisure.com>
ernstp pushed a commit to ernstp/curl that referenced this issue Mar 2, 2020
With ares the dns parameters lives in ares_channel. Store them in the
curl handle so we can set them again if we do easy_duphandle.

Closes curl#4893

Signed-off-by: Ernst Sjöstrand <ernst.sjostrand@verisure.com>
ernstp pushed a commit to ernstp/curl that referenced this issue Mar 6, 2020
With ares the dns parameters lives in ares_channel. Store them in the
curl handle so we can set them again if we do easy_duphandle.

Closes curl#4893

Signed-off-by: Ernst Sjöstrand <ernst.sjostrand@verisure.com>
@bagder bagder closed this in c8f086b Mar 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.