Implementing a Torznab indexer

kaso17 edited this page Oct 20, 2016 · 18 revisions

Introduction

Sonarr supports a Newznab-like api dubbed Torznab. This api offers a standardized recent/search api for both tv and movies (of which Sonarr only uses tv).
The Newznab api specification is built around a simple rss feed with filtering and paging capabilities. It was designed for usenet primarily, but it can be easily adapted for torrents.

The purpose of this wiki page is to describe the differences with Newznab and some recommendations to implement a custom proxy/indexer.

Private Trackers

Many torrent trackers thrive on the community and may have rules in place that mandate site visits, karma, votes, comments and all. Having an API doesn't need to be detrimental to this goal. A site can integrate the api into the existing karma/bonus system to keep users engaged in the community. There are quite a few creative ways of doing this.

Note: If your torrent tracker is based on a generic tracker software, please coordinate to get the api implemented in the generic base code so everyone can benefit and to avoid double work.

Differences with Newznab

Torznab extends on Newznab on a couple of points to make it more useful. However, it is in principle backward compatible.

Newznab/Torznab queries

Originally Newznab defined the query params q,rid,season,ep for t=tvsearch and q for t=search, however not all sites support tvrage ids, and therefore would have to be queries with the q= param. In later versions, Newznab returns a list of the supported query params in the t=caps response via a supportedParams attribute.

Sites should specify the parameters they support using supportedParams, allowing clients to perform the most optimal query, limiting the number of api calls and reducing database load. Clients should default supportedParams to q,rid,season,ep if a site doesn't provide it, thus remaining backward compatible. Clients should also query by *id as much as possible to avoid database keyword searches.

Multiple *id query params can be specified simultaneously (of which only one needs to match, so a OR operation on the db side).

t=tvsearch:
   Default query params: q,rid,season,ep
   Known query params  : q,rid,tvdbid,tvmazeid,season,ep
t=search:
   Default query params: q
   Known query params  : q
t=movie:
   Default query params: q
   Known query params  : q,imdbid

Examples:

/api?t=tvsearch&cat=..,..&rid=..&season=1&ep=1
/api?t=tvsearch&cat=..,..&q=..&season=1&ep=1

/api?t=tvsearch&cat=..,..&rid=..&season=2016&ep=12/20
/api?t=tvsearch&cat=..,..&q=..&season=2016&ep=12/20

/api?t=search&cat=..,..&q=Anime+Title+123
/api?t=search&cat=..,..&q=Fantastic+Series+With+Episode+Title+For+Special

/api?t=tvsearch&cat=..,..&rid=143&tvdbid=1232&season=1&ep=2 should perform a database query like SELECT * FROM Releases WHERE (rid=143 OR tvdbid=1232) AND season=1 AND ep=2. So q AND (rid OR tvdbid OR tvmazeid) AND season AND ep.

Caps endpoint

The mode t=caps is supposed to return the capabilities and categories of the newznab/torznab indexer. Implementing this mode is mandatory.

The caps response for torznab is mostly the same as newznab. However, newznab specifies a list of default categories. It's recommended to keep those categories intact, Client applications can then query one of the defaults, instea

Many torrent sites have their own rss categories starting at '1', it's recommended not to map these 1 to 1 on newznab categories but instead add 100000 to the number. That way it falls in the site specific range and does not interfere with the original newznab categories.
The api should then alias all site-specific categories in one of the original newznab category.

For example, if you have category 10 as Blu-Ray 1080p TV, then releases in that category would use the torznab api cats 100010,5040,5000 (Site-specific, TV/HD, TV). Any query that includes any of those numbers in the cat= parameter would return those releases as results.
Of course it's also allowed to omit the site-specific categories entirely, and map only on newznab categories.
A full list of default categories can be found in the References at the bottom of this page.

Example t=caps response:

      <?xml version="1.0" encoding="UTF-8"?>
         <caps>
            <!-- server information -->
            <server version="1.0" title="site" strapline="..."
             email="info@site.com" url="http://servername.com/" image="http://servername.com/theme/black/images/banner.jpg"/>

            <limits max="100" default="50"/>

            <registration available="yes" open="yes" />

            <searching>
               <search available="yes" supportedParams="q" />
               <tv-search available="yes" supportedParams="q,rid,tvmazeid,season,ep" />
               <movie-search available="no" supportedParams="q" />
            </searching>

            <!-- supported categories -->
            <categories>
               <category id="1000" name="Console">
                  <subcat id="1010" name="NDS"/>
                  <subcat id="1020" name="PSP"/>
               </category>
               <category id="2000" name="Movies">
                  <subcat id="2010" name="Foreign"/>
               </category>

               <!-- site specific categories -->
               <category id="100001" name="MotoGP"       description="Latest MotoGP stuff"/>
               <category id="100002" name="Fifa 2010"    description="Fifa 2010 world cup">
                  <subcat id="100003" name="Fifa 2010 HD" description="HD stuff"/>
                  <subcat id="100004" name="Fifa 2010 SD" description="SD stuff"/>
               </category>
               <!-- etc.. -->
            </categories>
         </caps>
      </xml>

Torznab results

First and foremost: Torznab indexers should return application/x-bittorrent as enclosure type:

<enclosure url="https://yoursite.com/download.php?torrent=123&amp;passkey=123456"   
           length="1460985071" type="application/x-bittorrent" />

Torznab defines a couple of attributes using the xmlns:torznab="http://torznab.com/schemas/2015/feed" xml namespace.
I'm keeping the list in the reference linked below, but here is a table with the torrent-specific stuff:
<torznab:attr name="seeders" value="1"/>

All these attributes are optional, the bold ones are recommended. Please note that this list consists of the attributes added to torznab, please check out the basic newznab attributes as well.

Name Type Description
type string series,movie,music,book (if unknown just omit)
size integer size of the release in bytes
files integer Number of files
grabs integer Number of times item downloaded
tvdbid integer id for thetvdb.com
rageid integer id for tvrage.com
tvmazeid integer id for tvmaze.com
imdb integer id for imdb.com
bannerurl url Url to a banner image
infohash string Torrent infohash
magneturl url Magnet uri. Not relevant for private trackers.
seeders integer Number of active seeders. Omit if unknown!
leechers integer Number of active non-seeders. Omit if unknown!
peers integer Number of active peers (seeders+leechers).
seedtype string TBD Specifies which seed criteria must be met. Was going for 'ratio,seedtime,both' but afaik it's always 'either'
minimumratio double Specifies the minimum ratio the torrent client must seed this torrent.
minimumseedtime integer Specifies the minimum number of seconds the torrent client must actively seed the torrent.
downloadvolumefactor double factor for the download volume, in most cases it should be set to 1, if a torrent is set to freeleech set it to 0, if only 50% is counted set it to 0.5
uploadvolumefactor double factor for the upload volume, in most cases it should be set to 1, if a torrent is set to neutral leech (upload is not counted) set it to 0, if it's set to double upload set it to 2

Minimum ratio/seedtime

These two attributes are included to allow a torrent tracker to specify the seeding requirements on a per torrent basis.
These values should be set to the long term requirements for the torrent. So if it's freeleech for a decade, you can set minimumratio to 0.0, but if it's only for a day, keep it at 1.0. (0.0 is a bit drastic, but you get the point)
The idea is that if one of the two criteria is satisfied, the torrent client can put the torrent at a lower priority or stop seeding entirely.

Specifying one or both of these attributes is recommended, because it allows future clients to appropriately adjust the seeding configuration of the torrent client and/or warn the user to avoid HnRs.

Hints and gotta's

  • Make sure your api properly handles the Accept and Content-Type http headers. Rss should have one of these mime-types application/rss+xml, text/rss+xml, text/xml. Whatever you do, do not use text/html.
  • Double check enclosure url, size and type.
  • Triple check any caching mechanism you have to make sure you're not serving stale or incorrect data.
  • Make sure that filter options you do not support return no results. Eg. if you don't support the 'q=' wildcard search, make sure you don't return any result if that query param is supplied.
  • Return the appropriate newznab errors on apilimits and such.
  • Support the offset= paging option.
  • Ensure you have a <guid>...</guid> item in your rss results.
  • Implement the t=caps endpoint, feel free to put it behind an apikey check.
  • Please only use the apikey for authentication, don't mandate cookies or other authentication.

Some References

Newznab specification: http://newznab.readthedocs.org/en/latest/misc/api/
nZEDb specification: https://github.com/nZEDb/nZEDb/blob/0.x/docs/newznab_api_specification.txt
Torznab xml attributes: https://github.com/Sonarr/Sonarr/blob/develop/schemas/torznab.xsd
Specific (older) example: https://github.com/Sonarr/Sonarr/blob/develop/src/NzbDrone.Core.Test/Files/Indexers/Torznab/torznab_hdaccess_net.xml