Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upHTTPS tracker timeouts cause 100% CPU usage #457
Comments
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
Jul 11, 2016
Here's a backtrace to the function trashing the CPU:
#0 0x00007fb92e117900 in PL_HashTableRawLookup () from target:/lib64/libplds4.so
#1 0x00007fb92e117d92 in PL_HashTableLookup () from target:/lib64/libplds4.so
#2 0x00007fb8e45dde82 in getUniquePEMNicknameFromFilename () from target:/lib64/libnsspem.so
#3 0x00007fb8e45e02dc in pem_CreateObject () from target:/lib64/libnsspem.so
#4 0x00007fb8e45e41a9 in nssCKFWSession_CreateObject () from target:/lib64/libnsspem.so
#5 0x00007fb8e45e8e0a in NSSCKFWC_CreateObject () from target:/lib64/libnsspem.so
#6 0x00007fb92e590262 in PK11_CreateNewObject () from target:/lib64/libnss3.so
#7 0x00007fb92e592033 in PK11_CreateGenericObject () from target:/lib64/libnss3.so
#8 0x00007fb930e97e77 in nss_create_object () from target:/lib64/libcurl.so.4
#9 0x00007fb930e97f32 in nss_load_cert () from target:/lib64/libcurl.so.4
#10 0x00007fb930eea39c in nss_connect_common () from target:/lib64/libcurl.so.4
#11 0x00007fb930ee741a in Curl_ssl_connect_nonblocking () from target:/lib64/libcurl.so.4
#12 0x00007fb930e9ea2d in Curl_http_connect () from target:/lib64/libcurl.so.4
#13 0x00007fb930eaff9c in Curl_protocol_connect () from target:/lib64/libcurl.so.4
#14 0x00007fb930ec430e in multi_runsingle () from target:/lib64/libcurl.so.4
#15 0x00007fb930ec5283 in multi_socket () from target:/lib64/libcurl.so.4
#16 0x00007fb930ec5407 in curl_multi_socket_action () from target:/lib64/libcurl.so.4
#17 0x00005620f3e246aa in core::CurlStack::receive_action(core::CurlSocket*, int) ()
#18 0x00007fb930baf1b4 in torrent::PollEPoll::perform() () from target:/lib64/libtorrent.so.19
#19 0x00007fb930baf2d1 in torrent::PollEPoll::do_poll(long, int) () from target:/lib64/libtorrent.so.19
#20 0x00007fb930bed71d in torrent::thread_base::event_loop(torrent::thread_base*) () from target:/lib64/libtorrent.so.19
#21 0x00005620f3d57d3d in main ()
oszi
commented
Jul 11, 2016
|
Here's a backtrace to the function trashing the CPU:
|
oszi
changed the title from
Mass HTTPS tracker timeouts kill rtorrent
to
Mass HTTPS tracker timeouts cause 100% CPU usage
Jul 11, 2016
oszi
changed the title from
Mass HTTPS tracker timeouts cause 100% CPU usage
to
HTTPS tracker timeouts cause 100% CPU usage
Jul 11, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Aug 13, 2016
Seeing this too. I found rtorrent using 100% CPU and the 12 torrents from a https tracker had a timeout error. I checked the tracker and it was running fine. Stoping the torrents made the 100% usage stop, but starting them made it appear again (note that tracker was OK at this time). After restarting rtorrent the torrents connected with the tracker without issue.
Edit:
Gentoo ~amd64, rtorrent 0.9.4, curl 7.50.1, libsigc++ 2.8.0, ncurses 6.0, xmlrpc-c 1.32.05, openssl 1.0.2h
ratpenat
commented
Aug 13, 2016
•
|
Seeing this too. I found rtorrent using 100% CPU and the 12 torrents from a https tracker had a timeout error. I checked the tracker and it was running fine. Stoping the torrents made the 100% usage stop, but starting them made it appear again (note that tracker was OK at this time). After restarting rtorrent the torrents connected with the tracker without issue. Edit: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
chros73
Aug 13, 2016
I checked the tracker and it was running fine.
What version, OS, libcurl, etc? :)
I also use one https tracker with around 200 torrents (for almost 2 years now), I never have this problem (and sometimes the tracker was unresponsive). I'm using rtorrent-ps v0.9.6 with pyroscope's build.sh script on Ubuntu 14.04.1. That build script takes care of this.
Oszi's problem probably different, though.
chros73
commented
Aug 13, 2016
What version, OS, libcurl, etc? :) I also use one https tracker with around 200 torrents (for almost 2 years now), I never have this problem (and sometimes the tracker was unresponsive). I'm using Oszi's problem probably different, though. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
Aug 13, 2016
I think it's the same thing. If I had to guess, there is a pool that processes tracker updates which fills up... If a tracker times out, there will be many requests waiting in the pool and it fills up quicker. But if you have a lot of torrents with https trackers (more than the pool's capacity), it will happen anyway.
The bug is definitely related to the libraries but it's possible rtorrent didn't catch up on some change.
(I switched to CentOS 7 + EPEL and everything is working there.)
oszi
commented
Aug 13, 2016
|
I think it's the same thing. If I had to guess, there is a pool that processes tracker updates which fills up... If a tracker times out, there will be many requests waiting in the pool and it fills up quicker. But if you have a lot of torrents with https trackers (more than the pool's capacity), it will happen anyway. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
chros73
Aug 14, 2016
I switched to CentOS 7 + EPEL and everything is working there.
Pretty interesting. :) What was the system before?
chros73
commented
Aug 14, 2016
Pretty interesting. :) What was the system before? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
commented
Aug 14, 2016
•
|
See my first post: Fedora 24 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Aug 14, 2016
Well, in my case it looks like it was because I was using curl 7.50.1 compiled with threads enabled instead of c-ares. 'curl-config --features' shows AsynchDNS in both cases.
An interesting side note, RES memory of rtorrent with ~5K torrents just after starting up and loading the torrents is very different in each case: 1.4 GiB with curl with c-ares versus 1 GiB with curl without c-ares and threads support enabled.
ratpenat
commented
Aug 14, 2016
•
|
Well, in my case it looks like it was because I was using curl 7.50.1 compiled with threads enabled instead of c-ares. 'curl-config --features' shows AsynchDNS in both cases. An interesting side note, RES memory of rtorrent with ~5K torrents just after starting up and loading the torrents is very different in each case: 1.4 GiB with curl with c-ares versus 1 GiB with curl without c-ares and threads support enabled. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
Aug 14, 2016
I don't see how would that only affect https trackers.
Can you check what is actually hogging your system? (I started with perf top.)
oszi
commented
Aug 14, 2016
|
I don't see how would that only affect https trackers. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Aug 15, 2016
Well, to be honest I don't know if this will happen with a HTTP tracker too. I haven't observed that case. Seems like my problem was different and I got fooled because I noticed it when a HTTPS tracker was down.
Your case looks more related with the ssl provider that curl is using, which can be one of axtls, gnutls, libressl, curl_ssl_mbedtls, nss, openssl, polarssl and winssl. I'm using curl compiled to use nss. I'll keep watching my CPU usage just in case.
ratpenat
commented
Aug 15, 2016
•
|
Well, to be honest I don't know if this will happen with a HTTP tracker too. I haven't observed that case. Seems like my problem was different and I got fooled because I noticed it when a HTTPS tracker was down. Your case looks more related with the ssl provider that curl is using, which can be one of axtls, gnutls, libressl, curl_ssl_mbedtls, nss, openssl, polarssl and winssl. I'm using curl compiled to use nss. I'll keep watching my CPU usage just in case. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Aug 20, 2016
OK, this happened again to me. I found 20+ HTTPS torrents from the same tracker with errors and rtorrent was using 100% CPU on one core for a few seconds while being unresponsive (interface not responding to key presses and net activity halted), then it would would get to work again (not getting response from the tracker for any of those torrents, but the tracker was OK) and after a few seconds the 100% CPU usage appears again and so on. If I stop those torrents rtorrent works fine again, but if I start just one of them hell braks loose again.
As oszi suggested, tried perf top and PL_HashTableRawLookup was producing 38% of all function calls. Sorry for no meaningful backtrace, my system is built without debug symbols and it's not a good time for a rebuild just now.
Looks like a library problem, but the fact that stopping a torrent and starting it again when the tracker is up still produces the issue might indicate that there is also something in rtorrent side.
I have:
rtorrent 0.9.6, libtorrent 0.13.6, libsigc++ 2.8.0, curl 7.50.1, xmlrpc-c 1.32.05, openssl 1.0.2h
Curl is built with c-ares 1.11.0 (the same happend if I used threaded support instead of ares) and with nss 3.20 as SSL provider. nss pulls in nspr 4.12
I've downgraded curl from 7.50.1 to 7.50.0 (what I had before seeing this issue) and so far so good. I'll keep watching.
ratpenat
commented
Aug 20, 2016
|
OK, this happened again to me. I found 20+ HTTPS torrents from the same tracker with errors and rtorrent was using 100% CPU on one core for a few seconds while being unresponsive (interface not responding to key presses and net activity halted), then it would would get to work again (not getting response from the tracker for any of those torrents, but the tracker was OK) and after a few seconds the 100% CPU usage appears again and so on. If I stop those torrents rtorrent works fine again, but if I start just one of them hell braks loose again. As oszi suggested, tried perf top and PL_HashTableRawLookup was producing 38% of all function calls. Sorry for no meaningful backtrace, my system is built without debug symbols and it's not a good time for a rebuild just now. Looks like a library problem, but the fact that stopping a torrent and starting it again when the tracker is up still produces the issue might indicate that there is also something in rtorrent side. I have: Curl is built with c-ares 1.11.0 (the same happend if I used threaded support instead of ares) and with nss 3.20 as SSL provider. nss pulls in nspr 4.12 I've downgraded curl from 7.50.1 to 7.50.0 (what I had before seeing this issue) and so far so good. I'll keep watching. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Aug 23, 2016
Happened again with the versions I listed in last post. Before trying to downgrade the next suspect (nss), I decided to try curl with openssl instead of nss. Problem seems gone. Also CPU usage is lower when a lot of torrents want to contact their tracker. Memory usage is algo slightly lower.
ratpenat
commented
Aug 23, 2016
|
Happened again with the versions I listed in last post. Before trying to downgrade the next suspect (nss), I decided to try curl with openssl instead of nss. Problem seems gone. Also CPU usage is lower when a lot of torrents want to contact their tracker. Memory usage is algo slightly lower. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Sep 5, 2016
After long testing I can confirm that curl compiled with openssl support is working fine. Problem is curl with nss.
ratpenat
commented
Sep 5, 2016
|
After long testing I can confirm that curl compiled with openssl support is working fine. Problem is curl with nss. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
commented
Sep 5, 2016
|
What an adventure. :D |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
chros73
Sep 6, 2016
curl compiled with openssl support is working fine
I think this is the default setting for curl. I wonder why did you used nss instead?
chros73
commented
Sep 6, 2016
I think this is the default setting for curl. I wonder why did you used nss instead? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ratpenat
Sep 6, 2016
The answer is the Gentoo package manager chose nss because I did not specifically chose one SSL provider for the curl package when I built the system a few weeks ago (there is no default choice). There are many packages that depend on curl installed with SSL support. In the definition of a package that is specified as I want curl with ssl and nss, or curl with ssl and openssl, or curl with ssl and libressl, etc. The first package that pulled curl had nss as the first option, so the package manager used that. Packages emerged later (libtorrent, for instance) find curl with ssl and nss and are OK with that (maybe rtorrent should have a dependency on curl with openssl). So, specifying I want curl with openssl and rebuilding the affected packages fixes that for me.
Edit: And by the way, Fedora 24 ships curl with nss (I read they have switched back and forth between nss and openssl) , which explains why oszi had the problem. I guess CentOS ships curl compiled with openssl and that's how he avoided the issue when he changed his setup.
ratpenat
commented
Sep 6, 2016
•
|
The answer is the Gentoo package manager chose nss because I did not specifically chose one SSL provider for the curl package when I built the system a few weeks ago (there is no default choice). There are many packages that depend on curl installed with SSL support. In the definition of a package that is specified as I want curl with ssl and nss, or curl with ssl and openssl, or curl with ssl and libressl, etc. The first package that pulled curl had nss as the first option, so the package manager used that. Packages emerged later (libtorrent, for instance) find curl with ssl and nss and are OK with that (maybe rtorrent should have a dependency on curl with openssl). So, specifying I want curl with openssl and rebuilding the affected packages fixes that for me. Edit: And by the way, Fedora 24 ships curl with nss (I read they have switched back and forth between nss and openssl) , which explains why oszi had the problem. I guess CentOS ships curl compiled with openssl and that's how he avoided the issue when he changed his setup. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
chros73
commented
Sep 7, 2016
|
Interesting :) Now it makes sense, thanks. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
oszi
commented
Sep 7, 2016
|
CentOS ships curl with nss too but it's only 7.29. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
smasher816
Dec 30, 2016
I'm hitting this issue on Arch linux. Their curl package relies on openssl but I'm not sure if it's using their ssl part. Is there a way to check? https://www.archlinux.org/packages/core/i686/curl/
smasher816
commented
Dec 30, 2016
|
I'm hitting this issue on Arch linux. Their curl package relies on openssl but I'm not sure if it's using their ssl part. Is there a way to check? https://www.archlinux.org/packages/core/i686/curl/ |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rezad1393
Dec 30, 2016
I too am on arch.
there was a new package update today for curl.
maybe test that?
UPDATE: updating to the new build of curl on archlinuxarm seems to have fixed the issue (https tracker timeouts)
rezad1393
commented
Dec 30, 2016
•
|
I too am on arch. UPDATE: updating to the new build of curl on archlinuxarm seems to have fixed the issue (https tracker timeouts) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
akiroz
Jan 19, 2017
Hmm.. updating to the new package on x86_64 didn't help.
I tried building curl from source WITH the new patch applied myself to make sure it wasn't using nss but still no luck.
curl version: 7.52.1
Host setup: x86_64-pc-linux-gnu
Install prefix: /usr
Compiler: gcc
SSL support: enabled (OpenSSL)
SSH support: enabled (libSSH2)
zlib support: enabled
GSS-API support: enabled (MIT Kerberos/Heimdal)
TLS-SRP support: enabled
resolver: POSIX threaded
IPv6 support: enabled
Unix sockets support: enabled
IDN support: no (--with-{libidn2,winidn})
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
SSPI support: no (--enable-sspi)
ca cert bundle: /etc/ssl/certs/ca-certificates.crt
ca cert path: no
ca fallback: no
LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support: no (--enable-ldaps)
RTSP support: enabled
RTMP support: no (--with-librtmp)
metalink support: no (--with-libmetalink)
PSL support: yes
HTTP2 support: disabled (--with-nghttp2)
Protocols: DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SCP SFTP SMB SMBS SMTP SMTPS TELNET TFTP
akiroz
commented
Jan 19, 2017
|
Hmm.. updating to the new package on x86_64 didn't help.
|
tavinus
referenced this issue
May 7, 2017
Closed
ddns-luci-app generates a 100% CPU utilization #3876
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Micdu70
Aug 12, 2017
Contributor
Hello, I have this issue with curl 7.55.0 (Debian 9), previous version (7.54.1) works perfectly...
Am I the only one?
EDIT: My results...
- curl 7.54.1 with OpenSSL (without c-ares) = OK
- curl 7.54.1 with OpenSSL (with c-ares) = High IO/100%
- curl 7.55.0 with OpenSSL (without c-ares) = High IO/100%
- curl 7.55.0 with OpenSSL (with c-ares) = High IO/100%
|
Hello, I have this issue with curl 7.55.0 (Debian 9), previous version (7.54.1) works perfectly... Am I the only one? EDIT: My results...
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
soredake
commented
Aug 20, 2017
|
@Micdu70 do you have torrent-file with https tracker to test? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
@soredake : I have only private trackers with https... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
chros73
Aug 20, 2017
You can give rtorrent-ps-ch a try maybe it will work fine (uses curl v7.54.1 with ssl and latest c-ares by default or you can specify older version).
chros73
commented
Aug 20, 2017
•
|
You can give rtorrent-ps-ch a try maybe it will work fine (uses curl v7.54.1 with ssl and latest c-ares by default or you can specify older version). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
inferiorhumanorgans
Aug 30, 2017
Yes this is an issue on FreeBSD 10 as well. curl 7.54.1 works, curl 7.55.1 causes 100% CPU usage.
inferiorhumanorgans
commented
Aug 30, 2017
|
Yes this is an issue on FreeBSD 10 as well. curl 7.54.1 works, curl 7.55.1 causes 100% CPU usage. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jay
Aug 30, 2017
Yes this is an issue on FreeBSD 10 as well. curl 7.54.1 works, curl 7.55.1 causes 100% CPU usage.
Hello I am a maintainer for the curl project and have been following this thread. AFAIK we have no similar reports with any other program. If you see a difference that is only in the curl library and it is reproducible would you mind running a bisect? https://github.com/curl/curl/wiki/how-to-git-bisect There aren't instructions for a cse like this but for example you could LD_PRELOAD the shared library after it's built.
jay
commented
Aug 30, 2017
Hello I am a maintainer for the curl project and have been following this thread. AFAIK we have no similar reports with any other program. If you see a difference that is only in the curl library and it is reproducible would you mind running a bisect? https://github.com/curl/curl/wiki/how-to-git-bisect There aren't instructions for a cse like this but for example you could LD_PRELOAD the shared library after it's built. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ss23
Sep 16, 2017
Contributor
This issue has returned for me after upgrading some packages on my system, including an upgrade to curl 7.55.1 also.
Unfortunately, I'm not in a position to do a bisect right now, but if I find more time I may be able to.
|
This issue has returned for me after upgrading some packages on my system, including an upgrade to curl 7.55.1 also. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ss23
Sep 16, 2017
Contributor
@jay I actually took some time and did a bisect:
5113ad0424044458ac497fa1458ebe0101356b22 is the first bad commit
commit 5113ad0424044458ac497fa1458ebe0101356b22
Author: Daniel Stenberg <daniel@haxx.se>
Date: Wed Jun 7 23:02:26 2017 +0200
http-proxy: do the HTTP CONNECT process entirely non-blocking
Mentioned as a problem since 2007 (8f87c15bdac63) and of course it
existed even before that.
Closes #1547
I don't know enough about cURL+rtorrent to know why this would cause an issue, but given the description, it seems like a reasonable candidate.
|
@jay I actually took some time and did a bisect:
I don't know enough about cURL+rtorrent to know why this would cause an issue, but given the description, it seems like a reasonable candidate. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
@ss23 : See #635 (comment) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ss23
Sep 16, 2017
Contributor
@Micdu70 That looks like the issue, guessing that solves it for the other people running 7.55 in here too, so not an issue with rtorrent.
I guess the fix is to revert to older curl and wait for a new release
|
@Micdu70 That looks like the issue, guessing that solves it for the other people running 7.55 in here too, so not an issue with rtorrent. I guess the fix is to revert to older curl and wait for a new release |
oszi commentedJul 11, 2016
•
edited
If an HTTPS tracker with a lot of torrents (100+) times out, rtorrent becomes completely unresponsive and the PL_HashTableRawLookup function in libplds4.so (called from libnss from curl) takes over 40% of all function calls in the system... This also causes a memory leak or it keeps allocating more.
Timeouts may not be the only reason but it escalates this madness. There is something really wrong with unresponsive HTTPS trackers. If a tracker responds with error messages, the same thing happens.
I tried to set
network.http.max_open.setto high values but rtorrent freezes even faster.Version: rtorrent-0.9.6-3.fc24.x86_64
Libs: curl-7.47.1-5.fc24.x86_64, nss-3.25.0-1.0.fc24.x86_64, nspr-4.12.0-1.fc24.x86_64