Skip to content

Load‐balancer

Jump to bottom
rgaudin edited this page Jun 26, 2026 · 4 revisions

Note

This is an internal doc about how the load-balancer is implemented. Please read Downloads first.

Our ZIM load-balancer is based on Mirrorbrain. We are still using mod_mirrorbrain and the MB database. All the tooling has been replaced though.

How it works?

A request is processed as follows:

  • https://lb.download.kiwix.org/ is first handled by the k8s ingress which adds CORS headers and handles / redirection.
  • lb-web deployment is a minibrain-server image that is mostly latest apache with mod_mirrorbrain and its dependencies (dbd, mod_form, mod_maxminddb).
  • mod_mirrorbrain checks if the file exists on disk. lb-web has access to fake data (see below).
  • It then checks if there's a match for this file on the DB.
  • If there is, it build a sorted list of online mirrors holding the file while capable of serving the file to this user (based on IP and IP location).
  • If it's a MB endpoint (digest, meta4, etc), it handles it with DB info.
  • If it's a standard request to the file, it redirects to the first mirror in the list.

Mirrorbrain Config

# prevents scanning non-zim folder, especially on mirrors which have (now obsolete) release and other folders
scan_top_include = zim
# size of chunks for SHA1. Smaller chunks means larger torrent/metalkink files
chunk_size = 4194304
MirrorBrainEngine On
# very useful when debugging but very verbose
MirrorBrainDebug Off
# very important as it ensures mb never attempts to send the file itself (required for us without real files)
MirrorBrainFallback "eu" "fr" "https://mirror.download.kiwix.org/"
# means that any file larger than this must be handled by mirrors (required for us without real files)
MirrorBrainMinSize 0
# Not required as we have the metadata but saves us some resources
MirrorBrainHandleHEADRequestLocally Off

Mirrorbrain DB

  • lb-db-sts
  • Runs on SRV-S
  • Latest postgres (18.4)
  • Started from a clean DB (using new copydb script)
  • New mirr_add_bypath_noins PLSQL function to record a file on a mirror

There are 3 important tables:

Server
id 2
identifier mirror.download.kiwix.org
enabled true
status_baseurl true
baseurl https://mirror.download.kiwix.org
country eu
country fr
only_region false
only_country true
other_countries it,gb,pt
Filearr
id 6590
path zim/wikipedia/wikipedia_fr_all_maxi_2026-05.zim
mirrors [2, 4, 7, 3]
Hash
file_id 6590
size 55438794004
mtime 1779688887
md5 b83b918bb2d0fb2cd2f5778219fa67b9
sha1 b25638806dc0f39682270822836a203e3e181094
sha256 61401b1b3618afe736450a868c7205f572f590db310e044a9c72b51fbfcfb431
btih a8e6e34f430c09af02e5ab852b6afc255b863a73

Cleanup

Every day (cronjob) we clean-up the database by removing all files that are not on our master mirror and triggering a VACUUM.

Probe

We probe mirrors to check their responsiveness.

  • mod_mirrorbrain only considers mirrors that have status_baseurl=True.
  • we thus test frequently that mirrors are online and change status_baseurl based on response.
  • this probe fails if it timesout after 10s (configurable)
  • it's a simple GET request (we only read headers)
  • we run this every minute

Scan

We scan mirrors to verify what files they hold.

  • We connect via rsync to get file listing
  • Wikimedia mirror is only one not giving us rsync access. We scan via HTTP 😕.
  • We update DB to remove the mirror ID from Filearr.mirrors for each file which is not present anymore and add it to all those present.
  • ⚠️ We differ from MirrorBrain which creates Filearr entries for every file found on mirrors. We don't. We ignore files not already in DB. Not only does this prevent DB from being polluted, it also allows us to control when a file is exposed in the LB.
  • We run this every mn, if the mirror is online.
  • When scanning, we don't consider mtime or size. That's MB behavior, possibly because HTTP scanning is not precise enough. We could maybe improve that if we get rid of HTTP.

Probe-then-scan Job

Because both probe and scan are run as frequently as every minute and because scan depends on probe, we:

  • have 1 cronjob per mirror (every minute, no concurrency). Allows customizing/suspending per mirror in the cluster.
  • run a forever loop script. That's to not have 10+ jobs created and deleted every mn.
  • run a probe-then-scan script that first probes and if successful runs the scan. Prevents scanning and log-pollution on probe failures
  • this takes 2-15s depending on mirrors.
  • Runs on SRV-S

Hashes

Hashes are used by mirrorbrain to respond to metadata requests such as digests. When querying the DB, mod_mirrobrain queries it with the mtime and size of the file on disk, ensuring that there's no discrepency between what it has access to and what it responds.

Important

For regular requests, the mtime and size are not considered.

When redirecting to a mirror, MB only ensures that this path was scanned on said mirror. The file could have disapeared on mirror or changed on mirror without the LB knowing.

Because we want to be torrent-first, we want hashes to be computed before offering a ZIM for download. That's why we are only creating Filearr entry when we have hashes ready. That differs from MB.

  • lb-makehashes cronjob (every minute, no concurrency).
  • Runs a forever script that triggers every mn.
  • Runs on SRV-B so it has access to actual ZIM files.
  • Filters only on *.zim to prevent adding non-ZIM files to DB.
  • Computes at about 100MiB/s. That's ~35mn for a 200GiB ZIM.

makehashes script works off two file trees: the actual ZIM files tree which will be used to compute hashes from and a fake hashes tree which will contain a copy of this file tree but composed entirely of sparse files: empty files faking actual size and mirrored mtime.

This hashes tree is used to figure which files have not had their hashes computed without hamering the DB: it's easy and fast to diff files on filesystem with their path, mtime and size.

A matching (path, mtime, size) file means it's up to date in DB.

# zim/wikipedia/wikipedia_el_all_nopic_2026-06.zim has its hashes computed and up-to-date
❯ ls -l /data/lb/zim/wikipedia/wikipedia_el_all_nopic_2026-06.zim
-rw-r--r--    1 root     root     1861146927 Jun 25 05:47 /data/lb/zim/wikipedia/wikipedia_el_all_nopic_2026-06.zim
❯ ls -l /data/hashes/zim/wikipedia/wikipedia_el_all_nopic_2026-06.zim
-rw-r--r--    1 root     root     1861146927 Jun 25 05:47 /data/hashes/zim/wikipedia/wikipedia_el_all_nopic_2026-06.zim

# ghana-bece-past-questions_en_all_2026-06.zim has been updated and needs new hashes
❯ ls -l /data/lb/zim/zimit/ghana-bece-past-questions_en_all_2026-06.zim
-rw-r--r--    1 root     root       8340798 Jun 16 04:32 /data/lb/zim/zimit/ghana-bece-past-questions_en_all_2026-06.zim
❯ ls -l /data/hashes/zim/zimit/ghana-bece-past-questions_en_all_2026-06.zim
-rw-r--r--    1 root     root      18340798 Jun 22 12:16 /data/hashes/zim/zimit/ghana-bece-past-questions_en_all_2026-06.zim

GeoIP

mod_mirrorbrain needs a working GeoIP system to retrieve country/region from IP.

We use the free City DB from Maxmind and update it daily in a cronjob running our geoipupdate image.

Mirrors list

We generate daily via a cronjob a mirrors.json file that replaces the old MB's mirrors.html. It is used by operations workflow tests to gather list of enabled mirrors.

Web server

mod_mirrorbrain needs access to served files on the filesystem to ensure they exist and retrieve both mtime and size when querying DB for hashes.

Because we don't want to rely on actual ZIM files (8+TiB), we maintain a copy of the entire repository but composed only of sparse files mirroring size and mtime. This lives in memory and is maintained in a fakesync side container that refreshes every 30s.

We also need to generate permalinks on the apache server (to ZIM and all MB endpoints). We do this in a side containers that stores it in-memory. It runs every minute.

Maintenance

           _       _ _               _
          (_)     (_) |             (_)
 _ __ ___  _ _ __  _| |__  _ __ __ _ _ _ __
| '_ ` _ \| | '_ \| | '_ \| '__/ _` | | '_ \
| | | | | | | | | | | |_) | | | (_| | | | | |
|_| |_| |_|_|_| |_|_|_.__/|_|  \__,_|_|_| |_|

MB conf at /etc/mirrobrain.conf
Available tools:
$ status            quick overview of mirror statuses
$ probe             check if a mirror is online or not
$ scan              scan a mirror's filetree to update DB with list of files on mirror
$ probe-then-scan   probe then scan if alive (shortcut for prod jobs)
$ makehashes        compute hashes from source files and update DB
$ test-hash         hash a single file and print its results
$ gen-zim-map       create an apache map of version-less ZIM filename to last path
$ cleandb           allows vacuuming the DB or removing gone (from master) files
$ genlist           generate list of mirrors (JSON format)
$ fileinfo          get all details about a specific file
$ latest            get list of latest added files on the DB

For monitoring and maintenance, we have two deployments: lb-tools and lb-tools-with-files.

The first one runs on SRV-S and only has access to the DB while the second one runs on SRV-B and also has access to actual ZIM files (/data/lb) and hashes (/data/hashes).

Those pods run minibrain-tools which should cover most needs.

Any advanced operation or query to the DB will require opening a port-forward.

FAQ

Adding a new mirror

  • Add IP to whitelist in rsyncd-configs and restart master-rsyncd
  • Manually add Server entry.
  • Create a new probe-then-scan job

When is a ZIM visible in LB?

  • ZIM journey starts in the CMS
  • CMS moves files on SRV-B to the correct location
  • ZIM is added and visible in the catalog
  • then:
    • makehashes sees it and computes hashes.
    • rsync job from SRV-M copies it to master mirror
  • LB is able to serve it
  • mirrors syncs it at their own schedule

ATM, this works because all of this happens fairly quickly but ideally https://github.com/openzim/cms/issues/103 should ensure that we publish only once it's on x mirrors.

What if hashes are wrong in DB?

  • Connect to lb-tools-with-files
  • Remove hashes tree version of file
  • Wait for makehashes to recreate it (<1mn)
rm -f /data/hashes/zim/wikipedia/wikipedia_en_bad_maxi-2026-01.zim

What if master mirror goes down?

xx

Clone this wiki locally