macOS: FTP + DoH fails with '(15) Could not DOH-resolve: {IPv4-address}'

[vszakats]

I understand it's an odd combination, but bumped into it anyhow. Initially I was looking into why the macOS-bundled curl 7.64.1 is doing a Segmentation fault: 11 with these options. I could not repeat it with a custom build of the latest source tree, but found this issue instead. Looks like curl wants to resolve the server IP via DoH.

I did this

./curl -q -vv \
  --doh-url https://dns.cloudflare.com/dns-query \
  ftp://ftp.thp.uni-duisburg.de/pub/source/pstree-2.39.tar.gz

stderr

* Found bundle for host dns.cloudflare.com: 0x7fe83240bfa0 [serially]
* Server doesn't support multiplex (yet)
*   Trying 2606:4700::6810:85e5:443...
* Hostname 'dns.cloudflare.com' was found in DNS cache
*   Trying 2606:4700::6810:85e5:443...
* Connected to dns.cloudflare.com (2606:4700::6810:85e5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Connected to dns.cloudflare.com (2606:4700::6810:85e5) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: cloudflare.com
* Server certificate: CloudFlare Inc ECC CA-2
* Server certificate: Baltimore CyberTrust Root
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe83281d600)
> POST /dns-query HTTP/2
Host: dns.cloudflare.com
accept: */*
content-type: application/dns-message
content-length: 41

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
* We are completely uploaded and fine
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: cloudflare.com
* Server certificate: CloudFlare Inc ECC CA-2
* Server certificate: Baltimore CyberTrust Root
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe832823200)
> POST /dns-query HTTP/2
Host: dns.cloudflare.com
accept: */*
content-type: application/dns-message
content-length: 41

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
* We are completely uploaded and fine
< HTTP/2 200 
< date: Mon, 25 May 2020 20:51:07 GMT
< content-type: application/dns-message
< content-length: 98
< access-control-allow-origin: *
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 599221f3bbf8ceab-GVA
< cf-request-id: 02ef338c560000ceabe3906200000001
< 
* Connection #0 to host dns.cloudflare.com left intact
* a DOH request is completed, 1 to go
< HTTP/2 200 
< date: Mon, 25 May 2020 20:51:07 GMT
< content-type: application/dns-message
< content-length: 123
< access-control-allow-origin: *
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 599221f3cc06ceab-GVA
< cf-request-id: 02ef338c5c0000ceabe3907200000001
< 
* Connection #1 to host dns.cloudflare.com left intact
* a DOH request is completed, 0 to go
* DOH Host name: ftp.thp.uni-duisburg.de
* TTL: 169695 seconds
* DOH A: 134.91.141.1
* CNAME: op2.thp.uni-duisburg.de
* CNAME: op2.thp.uni-duisburg.de
*   Trying 134.91.141.1:21...
* Connected to ftp.thp.uni-duisburg.de (134.91.141.1) port 21 (#0)
< 220 (vsFTPd 2.0.3)
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD pub
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD source
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||53657|)
* Could not DOH-resolve: 134.91.141.1
* Can't resolve new host 134.91.141.1:53657
* Closing connection 0
curl: (15) Could not DOH-resolve: 134.91.141.1

I expected the following

Start downloading the file at the URL while using the specified DoH resolver.

Omitting --doh-url fixes it. Adding --ftp-pasv doesn't.

curl/libcurl version: commit

curl 7.71.0-DEV (x86_64-apple-darwin19.4.0) libcurl/7.71.0-DEV SecureTransport zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) nghttp2/1.40.0 librtmp/2.3
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli HTTP2 IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL UnixSockets

operating system

macOS 10.15.4

[bagder]

I believe this is a macOS-specific issue because of this logic:

curl/lib/hostip.c

Lines 530 to 544 in 96a822f

	#ifndef USE_RESOLVE_ON_IPS
	/* First check if this is an IPv4 address string */
	if(Curl_inet_pton(AF_INET, hostname, &in) > 0)
	/* This is a dotted IP address 123.123.123.123-style */
	addr = Curl_ip2addr(AF_INET, &in, hostname, port);
	#ifdef ENABLE_IPV6
	if(!addr) {
	struct in6_addr in6;
	/* check if this is an IPv6 address string */
	if(Curl_inet_pton(AF_INET6, hostname, &in6) > 0)
	/* This is an IPv6 address literal */
	addr = Curl_ip2addr(AF_INET6, &in6, hostname, port);
	}
	#endif /* ENABLE_IPV6 */
	#endif /* !USE_RESOLVE_ON_IPS */

USE_RESOLVE_ON_IPS is defined in curl_setup.h if __APPLE is defined because we want to call getaddrinfo() even on plain IP addresses there. The logic just isn't adjusted to avoid DoH in this case!

[bagder]

Let me know how that PR works for you!