Skip to content
Searching torrents on popular trackers - CLI, readline, GUI, web client. Tutorial and binaries (issue tracker on https://gitlab.com/vindarel/cl-torrents/)
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets
src
tests
.gitattributes ignore html assets for github's project language recognition Nov 23, 2017
.gitignore
.gitlab-ci.yml CI: build gui Mar 16, 2019
.torrents.lisp
Makefile make install: clone nodgui (needs latest changes) Mar 18, 2019
README.org
index.org
theme-readtheorg.setup
torrents-test.asd
torrents.asd add description Mar 18, 2019
tutorial.org
version.lisp-expr

README.org

https://gitlab.com/vindarel/cl-torrents/badges/master/pipeline.svg

This is a little app to search for torrents on popular trackers.

It comes as:

  • a lisp library,
  • a command line interface,
  • a terminal readline interface,
  • a simple GUI.
  • The Web UI works too.

    We currently scrape 1337x.to (since v0.9) and torrentdownloads.me (v0.10) and present the results sorted by seeders.

    Download for GNU/Linux 64 bits: see Releases (78Mo, self-contained executable).

Installation

The library is on Quicklisp (february 2019) and Ultralisp.

(ql:quickload "torrents")
(torrents:search-torrents "tears of steel")

See the download link of the binary above. It is a self-contained executable for GNU/Linux 64-bits. You do not need to install a Lisp implementation. It’s a 78Mo binary.

To build the readline app, do:

make build

To build the Tk GUI:

make build-gui

then run

./torrents --help
./torrents-tk &

Troubleshooting

The Tk GUI needs nodgui newer than march, 2019.

If a search doesn’t work with a CRYPTO_num_locks error message:

You might need to upgrade your OpenSSL version. See this issue.

Usage

We can use this little app both in a Lisp REPL (Slime) and in a terminal, as a command line tool or from its embedded interactive prompt.

Results are cached in ~/.cl-torrents/cache/. Delete this directory if you want new results.

Lisp REPL

Search for torrents:

(use-package :torrents)
(search-torrents "matrix")
  [...]
  6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng]
  5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438
  4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind
  3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3
  2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du
  1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa
  0: Matrix FRENCH DVDRIP 1999 COOL

To help finding the most relevant results (the website sometimes is a bit relaxed with our search terms), each keyword is printed in a different color:

(you need to enable ansi colors in Slime, see slime-repl-ansi-color. Disable this feature with (setf cl-ansi-text:*enabled* nil))

We get a magnet link with:

(magnet 0)
;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"

We can open it with a torrent client (transmission-gtk by default):

(download 0)

And voilà :)

We may add more features, eventually. Our goal is rather to write a tutorial to show you diverse Common Lisp topics.

Parameters

*cache-p*: if nil, don’t use the cache system.

Command line

Use the options given below,

$ torrents -h
CL-torrents. Usage:

Available options:
  -d, --details            print more details (like the torrent's url)
  -h, --help               print this help text
  -i, --interactive        enter an interactive repl
  -m, --magnet ARG         get the magnet link of the given search result.
  -n, --nb ARG             set the maximum number of results to print.
  -o, --open INDEX         download with a torrent client (transmission-gtk by default)
  -v, --version            print the version

Example: below we search for “matrix…”, we display only 1 result and we get the magnet link of the first result.

$ torrents -n 1 matrix french 1999 cool -m 0

Readline interface

Start it with torrents -i.

See the help and the help of each command.

The available commands are (use the auto-completion):

  • search: search for torrents, print the results sorted by number of seeders.
  • magnet <i>, where i is a result index,
  • open or firefox <i>: open the given torrent page in a web browser
  • download <i>: open a magnet link with a torrent client (transmission-gtk by default)
  • filter <text>: show results that have this text in their title. This reduces the tab-completion of ids for all commands.
  • highlight (and unhighlight): highlight the given words in yellow, for this session.
  • url <i>
  • nb-results n to set the number of results to print to n
  • details: toggle the display of details
  • quit (or C-c)

And the available settings to change with set are (use set’s autocompletion):

  • *details*: if true, print more details below each row (like the url). Defaults to false.
  • *nb-results*: how many results to print. Defaults to 20.
  • *browser*: the browser name to use to open urls. Defaults to Firefox.
  • *torrent-client*: the desktop torrent client name. Defaults to Transmission.
  • *cache-p*: if true (the default), use the cached results.

You can also set them in a configuration file.

Note that we are in a regular readline interface, so the usual keybindings and options are available.

Configuration files (in development)

cl-torrents will read two configuration files. An ini-style one, and a lispy one.

Ini-style config

First, it will search for a ~.torrents.conf file from ~/.config/.torrents.conf and ~/.torrents.conf. The last one takes precedence.

For example, those are the default values:

[default]
scrapers = 1337 DOWNLOADSME
browser = firefox
nb-results = 20
cache-p = true  # use the cache system.

Moreover, you can set parameters related to the repl:

[default]
# again, default values:
confirm-exit = true  # ask with a yes/no prompt before quiting the app.
verbose = false
history = true  # use the repl history.
write-history = true  # read the history, but don't write to it.

You can set them for all replic-based apps in ~/.replic.conf, or override them in .torrents.conf.

Note: we only use a “default” profile for now.

Lisp init file

If the file ~/.torrents.lisp exists, it will be loaded after the .conf one and before the command line arguments.

The option --no-userinit prevents it from loading.

You can write whatever you want there, it is just a lisp file that will be load‘ed at startup.

Note: what we will eventually do then is to expose cl-torrents’ mechanism via hooks.

One thing you can do is define additional commands to be found at the repl. We do so by following replic’s mechanism:

  • define functions inside the torrents.user package
  • define the completion choices for some functions
  • export the functions you want to see turned into commands.

You can copy the example below in ~/.torrents.lisp and re-run torrents -i to try that out.

(in-package :torrents.user)

(defparameter *names* '()
  "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.")

(defun hello (name)
  "Takes only one argument. Adds the given name to the global
  `*names*` global variable, used to complete arguments of `goodbye`. "
  (format t "hello ~a~&" name)
  (push name *names*))

(defun goodbye (name)
  "Says goodbye to name, where `name` should be completed from what was given to `hello`."
  (format t "goodbye ~a~&" name))

;; Custom completion for goodbye:
(replic.completion:add-completion "goodbye" (lambda () *names*))

;; and export the two functions to find them as commands.
(export '(hello
          goodbye))

Ideas, todos

  • [X] cache (on files) v0.3
  • [X] CI
  • local copy of TPB
  • more command line options (specially verbosity)
    • [X] -i print url
  • [X] build with ECL. Used the lisp-devel docker image. Result: a 52Mo executable with a runtime error.
  • [X] open an url
  • download the torrent file
  • [X] ini config file, to parse with py-configparser
  • [X] add torrent to a local torrent client
  • add the torrent to a remote transmission client, with cl-transmission.
  • self-contained web app (doc),…
  • [X] interactive prompt, with completion of commands.

Release notes

https://gitlab.com/vindarel/cl-torrents/tags

dev (upcoming v0.11)

  • added: a simple Tk GUI (search, open in browser, download with torrent desktop client).
  • added: a filter command, to only display results whose title contains a given string. It reduces the TAB-completion of ids (but doesn’t constrain it).
  • added: all functions can now TAB-complete the list of ids.
  • added: scrape and display torrents’ size.
  • fixed -d cli arg.
  • added: load ~/.torrents.lisp, create new commands from the torrents.user package.
  • added the ability to read an ini-like config file, ~/.torrents.conf. All exported variables from the :torrent package can be overriden. They are written without earmuffs:
:*nb-results*
:*browser*
:*torrent-client*
:*cache-p*

gives

# ~/.torrents.conf
# those are the defaults.
[default]
nb-results = 20
browser = firefox
torrent-client = firefox
cache-p = true

See more on replic. (all parameters need more tests)

  • added missing -o option to download with a torrent client (transmission-gtk).
  • cl-torrents.conf configuration file:
    • choose the list of scrapers.
  • download command (open magnet link with transmission by default).
  • use the replic library to build the repl, the commands and their completions.
    • -> new builtin highlight and unhighlight commands,
    • -> better error handling (don’t exit the repl on an error).
  • fix cli usage without a keyword search, print help.

v0.10

v0.9

  • added 1337x.to scraper.
  • removed piratebay.to, katcr.co, torrent.cd, they’re down :/

v0.8

  • added a readline-based repl with some commands: search, magnet, open, with their completion. See help. Built with cl-readline, but might use replic anytime soon.

v0.7

  • fix cache not created by binary
  • create cache in ~/.cl-torrents/cache/

Dev

We use our replic library to automatically build the readline repl.

Clone cl-torrents in ~/quicklisp/local-projects,

now you can (ql:quickload :torrents).

Unit tests:

make test

End-to-end tests (checking that the websites respond and our scrapers still work):

make test-end2end

Resources

Don’t miss these good resources:

Tutorial

Writing a little web scraper like this one is not difficult. However, I had to spend some time to find out the right libraries and resources. It is also not trivial at first to start a Lisp project. So the first thing we did is write a tutorial. It is a mix of hopefully useful stuff:

  • web scraping,
  • async web scraping,
  • trying out things at the REPL,
  • where to find documentation,
  • creating and loading a project,
  • basic data structures and gotchas,
  • some useful libraries,
  • unit tests, with mocks, running tests from the shell, continuous integration,
  • parsing command line arguments, creating executables, continuous delivery,
  • basics of error handling,
  • It will eventually tackle more topics and features (more settings, working with a local copy of TPB…) but in the meanwhile, read

    the tutorial !

    It was built with https://github.com/fniessen/org-html-themes.

Licence

MIT.

You can’t perform that action at this time.