Skip to content

[regression] rtorrent breaks by commit "extract_if_dead: follow-up to 54b201b48c90a" #3542

Closed

Description

@tholin

I don't know if this is a bug in curl or rtorrent.

The bittorrent client rtorrent uses libcurl for http(s) trackers. If commit bbae24c "extract_if_dead: follow-up to 54b201b" is present rtorrent will loop over epoll_wait() and use 100% cpu. The commit was found by bisecting. The symptoms are identical to this older curl bug that also affected rtorrent rakshasa/rtorrent#580 (comment). I'm not familiar with the rtorrent code or exactly how it's using libcurl.

To reproduce the problem use a linux system with curl-7.64.0 or above with rtorrent-0.9.7. As long as there are some torrents running rtorrent should start executing around 500,000 epoll_wait calls/s after a few minutes.

$ curl -V
curl 7.64.0 (x86_64-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.0.2q zlib/1.2.11
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: Largefile NTLM SSL libz TLS-SRP UnixSockets HTTPS-proxy

Activity

  1. bagder commented on Feb 8, 2019

    @bagder
    Member

    I would urge you to try to come up with an example source code that only uses libcurl to allow us to reproduce the problem and help out the debugging. I will not be productive to have us build and try to untangle a full rtorrent build for this purpose.

  2. tholin commented on Feb 9, 2019

    @tholin
    Author

    I don't know anything about the libcurl api, how it's supposed to be used or how rtorrent use it. Investigating would take a lot of time and since I already have a workaround it would not be productive for me.

    I wanted to at least let you know that commit broke a downstream project. I'll file a bug report to my distribution. Perhaps the maintainer there wants to investigate it.

  3. jay commented on Feb 10, 2019

    @jay
    Member

    @tholin can you try the patch at #3541 (comment)

  4. tholin commented on Feb 11, 2019

    @tholin
    Author

    @tholin can you try the patch at #3541 (comment)

    I applied the patch to curl-7.64.0 but it had no effect. Rtorrent still started epolling like before.

  5. jay commented on Feb 11, 2019

    @jay
    Member

    Ok. Patch worked for the other reporter and is in master now. I don't know what could be causing it. Are you using proxies by any chance? Can you try master and this:

    diff --git a/lib/url.c b/lib/url.c
    index bc47685..f24a650 100644
    --- a/lib/url.c
    +++ b/lib/url.c
    @@ -964,8 +964,10 @@ static bool extract_if_dead(struct connectdata *conn,
           /* The protocol has a special method for checking the state of the
              connection. Use it to check if the connection is dead. */
           unsigned int state;
    +      struct Curl_easy *olddata = conn->data;
           conn->data = data; /* use this transfer for now */
           state = conn->handler->connection_check(conn, CONNCHECK_ISDEAD);
    +      conn->data = olddata;
           dead = (state & CONNRESULT_DEAD);
         }
         else {
  6. tholin commented on Feb 11, 2019

    @tholin
    Author

    I first confirmed that the problem still exists in master (cac0e4a) and then I applied the patch but it had no effect.

    I'm not using any proxy. Here is my rtorrent.rc config if it helps:

    bind=0.0.0.0
    throttle.min_peers.normal.set = 60
    throttle.max_peers.normal.set = 150
    network.port_range.set = 8000-8000
    protocol.pex.set = yes
    directory.default.set = /mnt/virtual/torrent
    dht.mode.set = on
    dht.port.set = 6881
    schedule = watch_directory,10,10,load.start=/mnt/virtual/torrent/*.torrent
    group.seeding.ratio.enable=
    ratio.min.set=110
    session.path.set = /home/tholin/.rtorrent
    ipv4_filter.load = ~/.rtorrent_blocklist,unwanted
    file.prioritize_toc.set=1
    encoding.add = UTF-8
    
  7. jay commented on Feb 11, 2019

    @jay
    Member

    Ok. How about this:

    diff --git a/lib/url.c b/lib/url.c
    index bc47685..46c8fb5 100644
    --- a/lib/url.c
    +++ b/lib/url.c
    @@ -964,8 +964,10 @@ static bool extract_if_dead(struct connectdata *conn,
           /* The protocol has a special method for checking the state of the
              connection. Use it to check if the connection is dead. */
           unsigned int state;
    +      struct Curl_easy *olddata = conn->data;
           conn->data = data; /* use this transfer for now */
           state = conn->handler->connection_check(conn, CONNCHECK_ISDEAD);
    +      conn->data = olddata;
           dead = (state & CONNRESULT_DEAD);
         }
         else {
    @@ -994,7 +996,6 @@ struct prunedead {
     static int call_extract_if_dead(struct connectdata *conn, void *param)
     {
       struct prunedead *p = (struct prunedead *)param;
    -  conn->data = p->data; /* transfer to use for this check */
       if(extract_if_dead(conn, p->data)) {
         /* stop the iteration here, pass back the connection that was extracted */
         p->extracted = conn;
  8. tholin commented on Feb 11, 2019

    @tholin
    Author

    This latest patch on master works fine. I ran it for 45 min to make sure and there is no excessive epolling with it.

  9. jay commented on Feb 12, 2019

    @jay
    Member

    Thanks for testing, I turned the patch into #3559 for review.

  10. locked as resolved and limited conversation to collaborators on May 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions