Better mirrorbrain configuration

[rgaudin]

Current mirrorbrain configuration is visible at kiwix/mirrors-qa#2
Basically, our mirrors have region and country data set but each are serving all requests (no xxOnly option set). The only parameter we set is the score.

Mirror	Score
fau.de	5000
dotsrc.org	5000
your.org	5000
mirrorservice.org	5000
nluug.nl	3000
mirror.accum.se	3000
laotzu-mirror.accum.se	3000
md.mirrors.hacktegic.com	3000
wikimedia	3000
mirror.kiwix	500
mirror-sites-ca.mblibrary.info	100
mirror-sites-fr.mblibrary.info	100
isoc.org.il	100
mirror-sites-in.mblibrary.info	100

Mirrorbrain's configuration is limited. here's what we can set:

Option	Description
regionOnly	If true, only clients from the same region (continent) as the mirror are redirected to this mirror.
countryOnly	If true, only clients from the same country as the mirror are redirected to this mirror.
asOnly	If true, the mirror will only get requests from clients that are located within the same network autonomous system (using the value in asn).
prefixOnly	If true, the mirror will only get requests from clients that are located within the same network prefix using the value inn prefix).
otherCountries	List of other countries that should be sent to this mirror server. This overrides the country and region choice, and can be used to fine-tune mirror selection. The list of country IDs specified here is given in the form of comma-separated two-letter codes.
score	The score (priority) of the server. Higher scored servers are used more often than lower scored servers. Default is 100. A server with score=150 will be used more often than a server with score=50.

---

Assuming we know the speeds of download from countries to mirrors (and in general to mirrors), how would we want to configure mirrorbrain?

One possibility I see is to set otherCountries on some mirror to force some countries-originating requests to go to specific mirrors but leave some other mirrors as currently in order to be able to serve requests from the rest of the world.

Scores should obviously be adjusted based on performance.

---

FYI, MB's code works as follows:

1. if there's a record in memcache, reuse it (valid for 10mn – can be configured)
2. sets list of mirrors in same_prefix, same_as, same_country, same_region, others
3. for each list in this order (prefix, as, country, region, others), it sorts it
4. if there's mirror in the list, pick the first

The sorting of mirrors is based on the score which is weighted with (basic) geographical distance between client and mirror.

Other facts I found in source:

• ?country=xx redirects immediately to a mirror in the requested country (if there is one)
• ?newmirror redirects to a mirror different from the previously returned one
• mirror's country_code can be set to *. Prevents per-country preference.
• debug in conf prints selection details in logs

Oh and mirrorbits have similar configuration options except it also accepts per-mirror excludeCountries which is handy to exclude under-performing combinations.

[kelson42]

I propose:

• we ignore countries - at least at this stage - and work at the region level
• Per default all mirrors are region only
• Scoring should be made proportionally to there ability to deliver content quickly
• Kiwix mirror should be global with a low score, another mirror with all the content should be configured in the same way

[rgaudin]

This has been implemented (via kiwix/mirrors-qa@42e69e8) and MB seems to react appropriately but this is extremely difficult to test due to #251