Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Curl causes high load on Synology devices #12517

Closed
cliffordwhansen opened this issue Dec 14, 2023 · 10 comments
Closed

Curl causes high load on Synology devices #12517

cliffordwhansen opened this issue Dec 14, 2023 · 10 comments
Labels
name lookup DNS and related tech not-a-curl-bug This is not a bug in curl

Comments

@cliffordwhansen
Copy link

I did this

I tried to run curl in a docker image using the latest Alpine image on a Synology DS1813+ and it just hangs.

I expected the following

I expected to get data returned from the URL.

curl/libcurl version

curl 8.5.0 (x86_64-alpine-linux-musl) libcurl/8.5.0 OpenSSL/3.1.4 zlib/1.3 brotli/1.1.0 c-ares/1.22.1 libidn2/2.3.4 nghttp2/1.58.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

operating system

alpine 3.19.0

@cliffordwhansen
Copy link
Author

I originally thought it might be Alpine related so I created this: alpinelinux/docker-alpine#365

For completeness, I paste the details here:

As a test, I used the image standalone and got the same result

silenus [~]$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
Status: Image is up to date for alpine:latest
docker.io/library/alpine:latest
silenus [~]$ docker run --rm -ti alpine sh
/ # apk add --no-cache curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ca-certificates (20230506-r0)
(2/8) Installing brotli-libs (1.1.0-r1)
(3/8) Installing c-ares (1.22.1-r0)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.5.0-r0)
(8/8) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 23 packages
/ # time curl https://google.com
^CCommand terminated by signal 2
real	8m 3.83s
user	4m 52.19s
sys	2m 56.06s
/ # 

Some more information:

silenus [~]$ uname -a
Linux silenus 3.10.108 #42962 SMP Mon May 29 14:36:28 CST 2023 x86_64 GNU/Linux synology_cedarview_1813+
silenus [~]$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 54
model name	: Intel(R) Atom(TM) CPU D2701   @ 2.13GHz
stepping	: 1
microcode	: 0x10d
cpu MHz		: 2127.775
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm arat invpcid_single
bogomips	: 4255.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 54
model name	: Intel(R) Atom(TM) CPU D2701   @ 2.13GHz
stepping	: 1
microcode	: 0x10d
cpu MHz		: 2127.775
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm arat invpcid_single
bogomips	: 4255.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 54
model name	: Intel(R) Atom(TM) CPU D2701   @ 2.13GHz
stepping	: 1
microcode	: 0x10d
cpu MHz		: 2127.775
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm arat invpcid_single
bogomips	: 4255.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 54
model name	: Intel(R) Atom(TM) CPU D2701   @ 2.13GHz
stepping	: 1
microcode	: 0x10d
cpu MHz		: 2127.775
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm arat invpcid_single
bogomips	: 4255.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

silenus [~]$ cat /proc/meminfo
MemTotal:        4042088 kB
MemFree:          770824 kB
Buffers:           14240 kB
Cached:          2309948 kB
SwapCached:       103696 kB
Active:           705284 kB
Inactive:        1959340 kB
Active(anon):     108012 kB
Inactive(anon):   244380 kB
Active(file):     597272 kB
Inactive(file):  1714960 kB
Unevictable:        1424 kB
Mlocked:            1424 kB
SwapTotal:       4523956 kB
SwapFree:        3915308 kB
Dirty:               364 kB
Writeback:             0 kB
AnonPages:        291680 kB
Mapped:           136948 kB
Shmem:             10476 kB
Slab:             198588 kB
SReclaimable:     100832 kB
SUnreclaim:        97756 kB
KernelStack:       13632 kB
PageTables:        51712 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6545000 kB
Committed_AS:    4518008 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      279868 kB
VmallocChunk:   34359420112 kB
DirectMap4k:       16876 kB
DirectMap2M:     4165632 kB

@jay
Copy link
Member

jay commented Dec 14, 2023

Use -v for verbose mode.

@dfandrich
Copy link
Contributor

dfandrich commented Dec 14, 2023 via email

@cliffordwhansen
Copy link
Author

This is curl on the synology

silenus [~]$ curl -V
curl 7.79.1 (x86_64-pc-linux-gnu) libcurl/7.79.1 OpenSSL/1.1.1t zlib/1.2.11 c-ares/1.14.0 nghttp2/1.41.0
Release-Date: 2021-09-22
Protocols: http https 
Features: alt-svc AsynchDNS Debug HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TrackMemory UnixSockets
silenus [~]$ curl -v https://google.com
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
> GET / HTTP/2
> Host: google.com
> user-agent: curl/7.79.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 301 
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-kMcYkn4jfqd1_UyY2wpbVg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< date: Thu, 14 Dec 2023 07:20:02 GMT
< expires: Sat, 13 Jan 2024 07:20:02 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

Running in docker

silenus [~]$ docker run --rm -ti alpine sh
/ # apk add --no-cache curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ca-certificates (20230506-r0)
(2/8) Installing brotli-libs (1.1.0-r1)
(3/8) Installing c-ares (1.22.1-r0)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.5.0-r0)
(8/8) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 23 packages
/ # time curl -vvvv https://google.com
^CCommand terminated by signal 2
real	2m 28.42s
user	1m 23.51s
sys	0m 51.53s

I canceled the curl call after a few of minutes

To test that it is not firewall related I was able to use wget

/ # wget
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: wget [-cqS] [--spider] [-O FILE] [-o LOGFILE] [--header STR]
	[--post-data STR | --post-file FILE] [-Y on/off]
	[-P DIR] [-U AGENT] [-T SEC] URL...

Retrieve files via HTTP or FTP

	--spider	Only check URL existence: $? is 0 if exists
	--header STR	Add STR (of form 'header: value') to headers
	--post-data STR	Send STR using POST method
	--post-file FILE	Send FILE using POST method
	-c		Continue retrieval of aborted transfer
	-q		Quiet
	-P DIR		Save to DIR (default .)
	-S    		Show server response
	-T SEC		Network read timeout is SEC seconds
	-O FILE		Save to FILE ('-' for stdout)
	-o LOGFILE	Log messages to FILE
	-U STR		Use STR for User-Agent header
	-Y on/off	Use proxy
/ # wget https://google.com
Connecting to google.com (192.178.54.46:443)
Connecting to www.google.com (192.178.54.68:443)
saving to 'index.html'
index.html           100% |************************************************************************************************************| 19836  0:00:00 ETA
'index.html' saved

@dfandrich
Copy link
Contributor

dfandrich commented Dec 14, 2023 via email

@cliffordwhansen
Copy link
Author

Sorry for the confusion, I think the version you were looking at was the one on the actual device which is working. The version in the container is as follows:

/ # curl -V
curl 8.5.0 (x86_64-alpine-linux-musl) libcurl/8.5.0 OpenSSL/3.1.4 zlib/1.3 brotli/1.1.0 c-ares/1.22.1 libidn2/2.3.4 nghttp2/1.58.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

I have done the following:

/ # apk add --no-cache curl c-ares
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing c-ares (1.22.1-r0)
(2/8) Installing ca-certificates (20230506-r0)
(3/8) Installing brotli-libs (1.1.0-r1)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.5.0-r0)
(8/8) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 23 packages
/ # curl -v https://google.com

And getting the same result.

These packages are pretty new.
https://pkgs.alpinelinux.org/package/v3.19/main/x86_64/c-ares
https://pkgs.alpinelinux.org/package/v3.19/main/x86_64/curl

@dfandrich
Copy link
Contributor

dfandrich commented Dec 14, 2023 via email

@cliffordwhansen
Copy link
Author

I tried replicating this on my laptop and got the following, which is also strange but at least it doesn't get stuck:

[~]$ docker run --rm -ti alpine sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
661ff4d9561e: Already exists 
Digest: sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
Status: Downloaded newer image for alpine:latest
/ # apk add --no-cache curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ca-certificates (20230506-r0)
(2/8) Installing brotli-libs (1.1.0-r1)
(3/8) Installing c-ares (1.22.1-r0)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.5.0-r0)
(8/8) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 12 MiB in 23 packages
/ # curl https://google.com
curl: (7) Failed to connect to google.com port 443 after 2003 ms: Couldn't connect to server
/ # 

I have attached a copy of the output from the strace

it is getting stuck on

getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)
getrandom(0x7f7d95580d00, 256, 0)       = -1 ENOSYS (Function not implemented)

output.txt

@bagder
Copy link
Member

bagder commented Dec 14, 2023

getrandom(0x7f7d95580d00, 256, 0) = -1 ENOSYS (Function not implemented)

The getrandom man page explains ENOSYS like this:

ENOSYS The  glibc wrapper function for getrandom() determined that the underlying
kernel does not implement this system call.

But also:

getrandom() was introduced in Linux 3.17.

According to logs above, this runs on a 3.10 kernel.

I believe c-ares is the one using this call, not curl itself. You can verify this by trying a curl build that does not use c-ares.

@bagder
Copy link
Member

bagder commented Dec 14, 2023

This is now fixed in c-ares. Not a curl bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
name lookup DNS and related tech not-a-curl-bug This is not a bug in curl
Development

No branches or pull requests

4 participants