Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added new section about tuning libtorrent for special needs
git-svn-id: https://libtorrent.svn.sourceforge.net/svnroot/libtorrent/trunk@3552 a83610d8-ad2a-0410-a6ab-fc0612d85776
- Loading branch information
arvidn
committed
May 25, 2009
1 parent
b4a6ffd
commit 34b6969
Showing
7 changed files
with
456 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| <?xml version="1.0" encoding="utf-8" ?> | ||
| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | ||
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> | ||
| <head> | ||
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> | ||
| <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" /> | ||
| <title>libtorrent manual</title> | ||
| <meta name="author" content="Arvid Norberg, arvid@rasterbar.com" /> | ||
| <link rel="stylesheet" type="text/css" href="../../css/base.css" /> | ||
| <link rel="stylesheet" type="text/css" href="../../css/rst.css" /> | ||
| <link rel="stylesheet" href="style.css" type="text/css" /> | ||
| <style type="text/css"> | ||
| /* Hides from IE-mac \*/ | ||
| * html pre { height: 1%; } | ||
| /* End hide from IE-mac */ | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <div class="document" id="libtorrent-manual"> | ||
| <div id="container"> | ||
| <div id="headerNav"> | ||
| <ul> | ||
| <li class="first"><a href="/">Home</a></li> | ||
| <li><a href="../../products.html">Products</a></li> | ||
| <li><a href="../../contact.html">Contact</a></li> | ||
| </ul> | ||
| </div> | ||
| <div id="header"> | ||
| <h1><span>Rasterbar Software</span></h1> | ||
| <h2><span>Software developement and consulting</span></h2> | ||
| </div> | ||
| <div id="main"> | ||
| <h1 class="title">libtorrent manual</h1> | ||
| <table class="docinfo" frame="void" rules="none"> | ||
| <col class="docinfo-name" /> | ||
| <col class="docinfo-content" /> | ||
| <tbody valign="top"> | ||
| <tr><th class="docinfo-name">Author:</th> | ||
| <td>Arvid Norberg, <a class="last reference external" href="mailto:arvid@rasterbar.com">arvid@rasterbar.com</a></td></tr> | ||
| <tr><th class="docinfo-name">Version:</th> | ||
| <td>0.15.0</td></tr> | ||
| </tbody> | ||
| </table> | ||
| <div class="contents topic" id="table-of-contents"> | ||
| <p class="topic-title first">Table of contents</p> | ||
| <ul class="simple"> | ||
| <li><a class="reference internal" href="#tuning-libtorrent" id="id1">tuning libtorrent</a></li> | ||
| <li><a class="reference internal" href="#reducing-memory-footprint" id="id2">reducing memory footprint</a><ul> | ||
| <li><a class="reference internal" href="#disable-disk-cache" id="id3">disable disk cache</a></li> | ||
| <li><a class="reference internal" href="#remove-torrents" id="id4">remove torrents</a></li> | ||
| <li><a class="reference internal" href="#socket-buffer-sizes" id="id5">socket buffer sizes</a></li> | ||
| <li><a class="reference internal" href="#peer-list-size" id="id6">peer list size</a></li> | ||
| <li><a class="reference internal" href="#send-buffer-watermark" id="id7">send buffer watermark</a></li> | ||
| <li><a class="reference internal" href="#optimize-hashing-for-memory-usage" id="id8">optimize hashing for memory usage</a></li> | ||
| <li><a class="reference internal" href="#reduce-executable-size" id="id9">reduce executable size</a></li> | ||
| </ul> | ||
| </li> | ||
| <li><a class="reference internal" href="#play-nice-with-the-disk" id="id10">play nice with the disk</a></li> | ||
| <li><a class="reference internal" href="#benchmarking" id="id11">benchmarking</a><ul> | ||
| <li><a class="reference internal" href="#disk-cache-stats" id="id12">disk cache stats</a></li> | ||
| <li><a class="reference internal" href="#disk-access-stats" id="id13">disk access stats</a></li> | ||
| <li><a class="reference internal" href="#session-stats" id="id14">session stats</a></li> | ||
| </ul> | ||
| </li> | ||
| </ul> | ||
| </div> | ||
| <div class="section" id="tuning-libtorrent"> | ||
| <h1>tuning libtorrent</h1> | ||
| <p>libtorrent expose most constants used in the bittorrent engine for | ||
| customization through the <tt class="docutils literal"><span class="pre">session_settings</span></tt>. This makes it possible to | ||
| test and tweak the parameters for certain algorithms to make a client | ||
| that fits a wide range of needs. From low memory embedded devices to | ||
| servers seeding thousands of torrents. The default settings in libtorrent | ||
| are tuned for an end-user bittorrent client running on a normal desktop | ||
| computer.</p> | ||
| <p>This document describes techniques to benchmark libtorrent performance | ||
| and how parameters are likely to affect it.</p> | ||
| </div> | ||
| <div class="section" id="reducing-memory-footprint"> | ||
| <h1>reducing memory footprint</h1> | ||
| <p>These are things you can do to reduce the memory footprint of libtorrent. You get | ||
| some of this by basing your default <tt class="docutils literal"><span class="pre">session_settings</span></tt> on the <tt class="docutils literal"><span class="pre">min_memory_usage()</span></tt> | ||
| setting preset function.</p> | ||
| <p>Keep in mind that lowering memory usage will affect performance, always profile | ||
| and benchmark your settings to determine if it's worth the trade-off.</p> | ||
| <p>The typical buffer usage of libtorrent, for a single download, with the cache | ||
| size set to 256 blocks (256 * 16 kiB = 4 MiB) is:</p> | ||
| <pre class="literal-block"> | ||
| read cache: 149.5 (2392 kiB) | ||
| write cache: 89.5 (1432 kiB) | ||
| receive buffers: 6.5 (104 kiB) | ||
| send buffers: 4 (64 kiB) | ||
| hash temp: 0.5 (8 kiB) | ||
| </pre> | ||
| <p>The receive buffers is proportional to the number of connections we make, and is | ||
| limited by the total number of connections in the session (default is 200).</p> | ||
| <p>The send buffers is proportional to the number of upload slots that are allowed | ||
| in the session. The default is auto configured based on the observed upload rate.</p> | ||
| <p>The read and write cache can be controlled (see section below).</p> | ||
| <p>The "hash temp" entry size depends on whether or not hashing is optimized for | ||
| speed or memory usage. In this test run it was optimized for memory usage.</p> | ||
| <div class="section" id="disable-disk-cache"> | ||
| <h2>disable disk cache</h2> | ||
| <p>The bulk of the memory libtorrent will use is used for the disk cache. To save | ||
| the absolute most amount of memory, you can disable the the cache by setting | ||
| <tt class="docutils literal"><span class="pre">session_settings::cache_size</span></tt> to 0. You might want to consider using the cache | ||
| but just disable caching read operations. You do this by settings | ||
| <tt class="docutils literal"><span class="pre">session_settings::use_read_cache</span></tt> to false. This is the main factor in how much | ||
| memory will be used by the client. Keep in mind that you will degrade performance | ||
| by disabling the cache. You should benchmark the disk access in order to make an | ||
| informed trade-off.</p> | ||
| </div> | ||
| <div class="section" id="remove-torrents"> | ||
| <h2>remove torrents</h2> | ||
| <p>Torrents that have been added to libtorrent will inevitably use up memory, event | ||
| when it's paused. A paused torrent will not use any peer connection objects or | ||
| any send or receive buffers though. Any added torrent holds the entire .torrent | ||
| file in memory, it also remembers the entire list of peers that it's heard about | ||
| (which can be fairly long unless it's capped). It also retains information about | ||
| which blocks and pieces we have on disk, which can be significant for torrents | ||
| with many pieces.</p> | ||
| <p>If you need to minimize the memory footprint, consider removing torrents from | ||
| the session rather than pausing them. This will likely only make a difference | ||
| when you have a very large number of torrents in a session.</p> | ||
| <p>The downside of removing them is that they will no longer be auto-managed. Paused | ||
| auto managed torrents are scraped periodically, to determine which torrents are | ||
| in the greatest need of seeding, and libtorrent will prioritize to seed those.</p> | ||
| </div> | ||
| <div class="section" id="socket-buffer-sizes"> | ||
| <h2>socket buffer sizes</h2> | ||
| <p>You can make libtorrent explicitly set the kernel buffer sizes of all its peer | ||
| sockets. If you set this to a low number, you may see reduced throughput, especially | ||
| for high latency connections. It is however an oportunity to save memory per | ||
| connection, and might be worth considering if you have a very large number of | ||
| peer connections. This memory will not be visible in your process, this sets | ||
| the amount of kernel memory is used for your sockets.</p> | ||
| <p>Change this by setting <tt class="docutils literal"><span class="pre">session_settings::recv_socket_buffer_size</span></tt> and | ||
| <tt class="docutils literal"><span class="pre">session_settings::send_socket_buffer_size</span></tt>.</p> | ||
| </div> | ||
| <div class="section" id="peer-list-size"> | ||
| <h2>peer list size</h2> | ||
| <p>The default maximum for the peer list is 4000 peers. For IPv4 peers, each peer | ||
| entry uses 32 bytes, which ends up using 128 kB per torrent. If seeding 4 popular | ||
| torrents, the peer lists alone uses about half a megabyte.</p> | ||
| <p>The default limit is the same for paused torrents as well, so if you have a | ||
| large number of paused torrents (that are popular) it will be even more | ||
| significant.</p> | ||
| <p>If you're short of memory, you should consider lowering the limit. 500 is probably | ||
| enough. You can do this by setting <tt class="docutils literal"><span class="pre">session_settings::max_peerlist_size</span></tt> to | ||
| the max number of peers you want in the torrent's peer list.</p> | ||
| <p>You should also lower the same limit but for paused torrents. It might even make sense | ||
| to set that even lower, since you only need a few peers to start up while waiting | ||
| for the tracker and DHT to give you fresh ones. The max peer list size for paused | ||
| torrents is set by <tt class="docutils literal"><span class="pre">session_settings::max_paused_peerlist_size</span></tt>.</p> | ||
| <p>The drawback of lowering this number is that if you end up in a position where | ||
| the tracker is down for an extended period of time, your only hope of finding live | ||
| peers is to go through your list of all peers you've ever seen. Having a large | ||
| peer list will also help increase performance when starting up, since the torrent | ||
| can start connecting to peers in parallel with connecting to the tracker.</p> | ||
| </div> | ||
| <div class="section" id="send-buffer-watermark"> | ||
| <h2>send buffer watermark</h2> | ||
| <p>The send buffer watermark controls when libtorrent will ask the disk I/O thread | ||
| to read blocks from disk, and append it to a peer's send buffer.</p> | ||
| <p>When the send buffer has fewer than or equal number of bytes as | ||
| <tt class="docutils literal"><span class="pre">session_settings::send_buffer_watermark</span></tt>, the peer will ask the disk I/O thread | ||
| for more data to send. The trade-off here is between wasting memory by having too | ||
| much data in the send buffer, and hurting send rate by starving out the socket, | ||
| waiting for the disk read operation to complete.</p> | ||
| <p>If your main objective is memory usage and you're not concerned about being able | ||
| to achieve high send rates, you can set the watermark to 9 bytes. This will guarantee | ||
| that no more than a single (16 kiB) block will be on the send buffer at a time, for | ||
| all peers. This is the least amount of memory possible for the send buffer.</p> | ||
| <p>You should benchmark your max send rate when adjusting this setting. If you have | ||
| a very fast disk, you are less likely see a performance hit.</p> | ||
| </div> | ||
| <div class="section" id="optimize-hashing-for-memory-usage"> | ||
| <h2>optimize hashing for memory usage</h2> | ||
| <p>When libtorrent is doing hash checks of a file, or when it re-reads a piece that | ||
| was just completed to verify its hash, there are two options. The default one | ||
| is optimized for speed, which allocates buffers for the entire piece, reads in | ||
| the whole piece in one read call, then hashes it.</p> | ||
| <p>The second option is to optimize for memory usage instead, where a single buffer | ||
| is allocated, and the piece is read one block at a time, hashing it as each | ||
| block is read from the file. For low memory environments, this latter approach | ||
| is recommended. Change this by settings <tt class="docutils literal"><span class="pre">session_settings::optimize_hashing_for_speed</span></tt> | ||
| to false. This will significantly reduce peak memory usage, especially for | ||
| torrents with very large pieces.</p> | ||
| </div> | ||
| <div class="section" id="reduce-executable-size"> | ||
| <h2>reduce executable size</h2> | ||
| <p>Compilers generally add a significant number of bytes to executables that make use | ||
| of C++ exceptions. By disabling exceptions (-fno-exceptions on GCC), you can | ||
| reduce the executable size with up to 45%.</p> | ||
| <p>Also make sure to optimize for size when compiling.</p> | ||
| </div> | ||
| </div> | ||
| <div class="section" id="play-nice-with-the-disk"> | ||
| <h1>play nice with the disk</h1> | ||
| <p>When checking a torrent, libtorrent will try to read as fast as possible from the disk. | ||
| The only thing that might hold it back is a CPU that is slow at calculating SHA-1 hashes, | ||
| but typically the file checking is limited by disk read speed. Most operating systems | ||
| today do not prioritize disk access based on the importance of the operation, this means | ||
| that checking a torrent might delay other disk accesses, such as virtual memory swapping | ||
| or just loading file by other (interactive) applications.</p> | ||
| <p>In order to play nicer with the disk, and leave some spare time for it to service other | ||
| processes that might be of higher importance to the end-user, you can introduce a sleep | ||
| between the disc accesses. This is a direct tradeoff between how fast you can check a | ||
| torrent and how soft you will hit the disk.</p> | ||
| <p>You control this by setting the <tt class="docutils literal"><span class="pre">session_settings::file_checks_delay_per_block</span></tt> to greater | ||
| than zero. This number is the number of milliseconds to sleep between each read of 16 kiB.</p> | ||
| <p>The sleeps are not necessarily in between each 16 kiB block (it might be read in larger chunks), | ||
| but the number will be multiplied by the number of blocks that were read, to maintain the | ||
| same semantics.</p> | ||
| </div> | ||
| <div class="section" id="benchmarking"> | ||
| <h1>benchmarking</h1> | ||
| <div class="section" id="disk-cache-stats"> | ||
| <h2>disk cache stats</h2> | ||
| </div> | ||
| <div class="section" id="disk-access-stats"> | ||
| <h2>disk access stats</h2> | ||
| </div> | ||
| <div class="section" id="session-stats"> | ||
| <h2>session stats</h2> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div id="footer"> | ||
| <span>Copyright © 2005 Rasterbar Software.</span> | ||
| </div> | ||
| </div> | ||
| <script src="http://www.google-analytics.com/urchin.js" type="text/javascript"> | ||
| </script> | ||
| <script type="text/javascript"> | ||
| _uacct = "UA-1599045-1"; | ||
| urchinTracker(); | ||
| </script> | ||
| </div> | ||
| </body> | ||
| </html> |
Oops, something went wrong.