curl doesn't accept DNS responses on alpine:3.19

[NikitaSkrynnik]

I did this

Hello! I downloaded curl on alpine:3.19 with

apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ca-certificates (20230506-r0)
(2/8) Installing brotli-libs (1.1.0-r1)
(3/8) Installing c-ares (1.22.1-r0)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.5.0-r0)
(8/8) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 23 packages

A pod with apline is deployed in a Kubernetes cluster. I send a request to greeting.default service on the cluster and get this error:

curl greeting.default:9080
curl: (6) Could not resolve host: greeting.default

The problem is that if I do the request on apline:3.18 I get a correct response from the service.

curl greeting.default:9080
"hello world from istio"

I install curl on alpine:3.18 the same way:

apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
curl greetinfetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
g.default(1/7) Installing ca-certificates (20230506-r0)
(2/7) Installing brotli-libs (1.0.9-r14)
(3/7) Installing libunistring (1.1-r1)
(4/7) Installing libidn2 (2.3.4-r1)
(5/7) Installing nghttp2-libs (1.57.0-r0)
(6/7) Installing libcurl (8.5.0-r0)
(7/7) Installing curl (8.5.0-r0)
Executing busybox-1.36.0-r9.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 22 packages

We use our own DNS server on the pod with alpine and it uses search domains automatically. For example, if I do DNS lookup for greeting.service I can get greeting.default.svc.cluster.local:

dig greeting.default

; <<>> DiG 9.18.19 <<>> greeting.default
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50637
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: efc11b1d4d505bac (echoed)
;; QUESTION SECTION:
;greeting.default.		IN	A

;; ANSWER SECTION:
greeting.default.svc.cluster.local. 30 IN A	172.16.1.2

;; Query time: 96 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Wed Dec 20 07:26:50 UTC 2023
;; MSG SIZE  rcvd: 107

It looks like alpine:3.19's curl doesn't accept such DNS responses. I tried to manually change the DNS response from greeting.default.svc.cluster.local to greeting.default in ANSWER SECTION and curl (alpine:3.19) accepted that.

I tried ping and it accepted the greeting.default.svc.cluster.local response from the DNS server. curl (alpine:3.18) is also ok with this response.

ping greeting.default -c 4
PING greeting.default (172.16.1.2): 56 data bytes
64 bytes from 172.16.1.2: seq=0 ttl=62 time=0.389 ms
64 bytes from 172.16.1.2: seq=1 ttl=62 time=0.650 ms
64 bytes from 172.16.1.2: seq=2 ttl=62 time=0.845 ms
64 bytes from 172.16.1.2: seq=3 ttl=62 time=0.536 ms

--- greeting.default ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.389/0.605/0.845 ms

curl (alpine:3.19) doesn't accept this DNS response. Is it a bug? Or this behaviour has been changed intentionally?

I expected the following

The curl results on alpine:3.18 and alpine:3.19 are the same.

curl/libcurl version

alpine:3.18

curl --version
curl 8.5.0 (x86_64-alpine-linux-musl) libcurl/8.5.0 OpenSSL/3.1.0 zlib/1.2.13 brotli/1.0.9 libidn2/2.3.4 nghttp2/1.57.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

alpine:3.19

curl --version
curl 8.5.0 (x86_64-alpine-linux-musl) libcurl/8.5.0 OpenSSL/3.1.4 zlib/1.3 brotli/1.1.0 c-ares/1.22.1 libidn2/2.3.4 nghttp2/1.58.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

operating system

alpine:3.18

Linux alpine-77c998fc5b-7z48p 5.15.0-91-generic #101~20.04.1-Ubuntu SMP Thu Nov 16 14:22:28 UTC 2023 x86_64 Linux

apline:3.19

Linux alpine-6d5479b7fc-rpj88 5.15.0-91-generic #101~20.04.1-Ubuntu SMP Thu Nov 16 14:22:28 UTC 2023 x86_64 Linux

[dfandrich]

The alpine:3.19 curl is showing that it's using the c-ares resolver, while the alpine:3.18 version is using the system resolver. That's likely to be the reason for the change in behaviour. Is there any relevant option in /etc/resolv.conf (e.g. ndots) that could affect this?

[NikitaSkrynnik]

@dfandrich I tried different values for ndots (ndots=1, ndots=5, ndots=10) but the result is the same. I don't use any other options.

cat /etc/resolv.conf
#search ns-dns.svc.cluster.local svc.cluster.local cluster.local
#nameserver 10.96.0.10
#options ndots:5
#

options ndots:5
nameserver 127.0.0.1

[bagder]

This sounds like a (potential) c-ares issue.

[bagder]

If this problem remains with the latest c-ares version, I propose you file an issue in the @c-ares project.