This is a work in progress. Please do send patches and provide feedback. Thanks!
The project is similar to mirrorbrain (.org) and fedora's mirrors system. However, it has a few differences (this list is not intended to be complete):
- it is very specific to the way Debian mirrors are constructed. Details regarding architectures and the different mirror types are taken into consideration.
- because of the previous point and considering many mirrors only support http, it does not perform a full mirror scan. Mirrorbrain does. There's a tool to detect inconsistencies between what the mirrors master list claims a mirror contains and what it actually contains.
- it aims to be httpd-independent. Mirrorbrain requires apache.
- IPv6 support
- no DBMS. Although using a DBMS could provide some advantages, at the moment the Storable database seems to be enough. The idea is to keep everything simple.
- easy deployment
There's a live instance of this code (but not necessarily the latest and greatest revision) at http://httpredir.debian.org/ (previously, http.debian.net)
There's some more documentation there. It should be imported into the repository, however.
There are some TODOs and FIXMEs in the source code, additionally, there's the issues tracker at github: https://github.com/rgeissert/http-redirector/issues
libev-perl(recommended; or another event loop supported by AE)
liblinux-inotify2-perl(for plackup -R, for local tests only)
starman(if using that server for the application)
./update.sh, it will download the geoip databases, the mirrors
list, build the database used by the redirector, and check the mirrors
Look at the example below for invocation and plackup(1p) and Plack's documentation for running the application under different server modes.
If you just want to simulate a request (like it used to be possible
local-request.pl. It sends a request to the
application without actually starting the server.
You can pass request parameters in the first argument to the script.
To fake the IP address of the request, set the
REMOTE_ADDR env var when
calling it. No other CGI-like env var is recognised.
Getting started for development
Required packages (in addition to the ones above):
Keeping everything in shape
update.sh should be run at least once a month, this allows the changes to the mirror list(s) to be reflected. By default it will run:
check.pl should be run multiple times a day
build-peers-db.pl should be run after every execution of
build-main-db.pl or whenever the peers lists are updated
NOTE: update.sh will leave the new database in a file called
to be renamed to
db if it is the first time it is created. When
updating the db of an in-production instance, the new db will be picked up
by the next run of
check.pl do NOT lock the database. You must
ensure that no more than one script is running at the same time.
 the script rebuilds the database, so any info collected by check.pl regarding the availability of mirrors is lost. check.pl --check-everything should be run after build-main-db.pl, this is done automatically when running update.sh.
 it really depends on the kind of setup one wants and the hosts that conform the mirrors network. For Debian's archive, it should be run at least every ten minutes, every five minutes being better.
 it only applies when using an AS peers database. The name of the peers database is specific to the mirrors database on which it is based. At present, peers databases for old mirrors databases are not cleaned up automatically.
Real life testing
If using apache, you will want to run the redirector locally and make apache forward the requests (therefore acting as a reverse proxy). For example, if you run the application on port 5000 you can:
ProxyPass /redir http://127.0.0.1:5000/ RewriteEngine On RewriteRule ^/?(?:(demo)/)?debian-(security|backports|ports)/(.*) /redir/?mirror=$2&url=$3&action=$1 [PT] RewriteRule ^/?(?:(demo)/)?debian-archive/(.*) /redir/?mirror=old&url=$2&action=$1 [PT] RewriteRule ^/?(?:(demo)/)?debian/(.*) /redir/?mirror=archive&url=$2&action=$1 [PT] # mirror:// method support: RewriteRule ^/?debian-(security|backports|ports)\.list(?:$|\?(.+)) /redir/?mirror=$1.list$2 [QSA,PT] RewriteRule ^/?debian-archive\.list(?:$|\?(.+)) /redir/?mirror=old.list$1 [QSA,PT] RewriteRule ^/?debian\.list(?:$|\?(.+)) /redir/?mirror=archive.list$1 [QSA,PT]
You can for example make it listen on 127.0.1.10, setup a vhost, and use the following on your sources.list:
deb http://127.0.1.10/debian/ sid main deb-src http://127.0.1.10/debian/ sid main deb http://127.0.1.10/debian-security/ testing/updates main deb http://security.debian.org/ testing/updates main
Note: accessing the redirector from a local IP address is not ideal and may only work with hacks.
Forcibly disabling mirrors
If necessary, mirrors can forcibly be disabled by passing a file name
--disable-sites option (
The format of this file is:
<domain name>[/mirror type]
Whenever the option is passed and the file exists, every mirror matching an entry in the file will be disabled without further checks. If a mirror type is specified, only that mirror type of the given mirror will be disabled.
An empty file name can be specified to override the default and to prevent the parsing of said file. In order to re-enable a mirror an existing file name must be specified.
NOTE: any disabled mirror that is no longer in the list will be
re-enabled. E.g. passing
--disable-sites /dev/null to
re-enable all disabled mirrors.
Running the redirector on top of a real mirror
It is possible to run the redirector on a sever that has the files itself. The use case would be: serve some users, send others to a better mirror.
In order to use it in this mode, a few things need to be setup.
- Set the
subrequest_methodvariable as appropriate:
- redirect: works on any httpd, but requires another roundtrip
- sendfile: for apache with mod_xsendfile, lighttpd 1.5, cherokee
- sendfile1.4: for lighttpd 1.4
- accelredirect: for nginx
- Set the list of hosts for which the files will be served, in the
this_hostvariable. For example, if your host is
my.mirror.tld, that's what you need to add.
NOTE: even though it is possible to list other mirror's host names, care should be taken when doing so. The mirror checker is not aware of this mapping and may lead to erroneous behaviour.
NOTE2: make sure your httpd works as expected. The redirector sends a Content-Location header when the file should be delivered by the server itself. Make sure it is not removed. Also look for issues with Range, Last-Modified, and other features.
Then, setup your httpd so that requests for serve/ are served directly, bypassing the redirector. Finally, make all the usual traffic go through the redirector. Make sure you don't break directory listing when doing so. mod_xsendfile, for example, breaks it because it bypasses mod_autoindex.
A sample configuration for apache follows:
XSendFile On # Should be possible to do it without an alias, but it makes it a # bit clearer Alias /debian/serve/ "/var/www/debian/" RewriteEngine On # Exclude /debian/serve/ from mod_rewrite, mod_alias will handle it RewriteRule ^/?debian/serve/.*$ $0 [PT,L] # Send all file requests through the redirector RewriteRule ^/?debian/(.*[^/]$) /redir/?mirror=archive&url=$1 [NS,PT] # Directory listing requests will pass-through and be handled by # apache itself
IMPORTANT: do not enable this setup, or run the redirector at all, on a mirror that is part of Debian's mirrors network without consulting the mirroradmin group and WAITING for their APPROVAL.
AS peers database
It is possible to instruct the redirector to serve clients from an originating AS to one or a set of destination AS' where mirrors are located. The database can be built with the build-peers-db.pl script.
Its input are *.lst files in the peers.lst.d directory in the following format:
<client AS> <mirror domain name|mirror AS> [distance [IPv]]
The preferred form of the second value is by domain name. When an AS is specified, it will internally be rewritten to the existing mirrors in the corresponding AS.
Comments may be specified by prefixing them with a
Multiple client ASNs can be specified by enclosing them in braces and
separating them with commas. E.g.
The distance is currently not used by the redirector, but at some point it might. It defaults to 0. Any positive integer may be specified. It is expected to be read as 0 being the most preferred mirror, 1 being the second, 2 being the third, and so on.
The IPv field, if specified, should be
v6, or the two separated
by a comma with no specific order. It can be used to indicate that the
given peering rule only applies to the version of the IP specified in
the field. It defaults to
Mirrors chosen by this database are still subject to geo location restriction. I.e. from the set of candidates, a subset of those that are geographically closer will be created and used.
Caveats: since the database is AS-based, large Autonomous Systems that traverse countries or continents will still be considered, even if not desired. At present, the redirector skips mirrors that are located in a different continent, but that's done to work around another issue and the behaviour is not guaranteed to persist after the other issue is properly addressed.
Understanding the db
The database consists of (mostly inverted) indexes that are supposed to provide fast and cheap lookups.
In order to save space on the database, a few unusual things are done.
For example, hash entries with
undef as value are valid.
smaller in a Storable database than an integer.
Any script using the database should therefore test for 'exists' instead
To better understand what the database looks like, run ./dump-db.pl | pager
"This product includes GeoLite data created by MaxMind, available from http://maxmind.com/"