Download Wikipedia articles' summaries only / download separately from map files

[pastk]

"Summary" is an article's part before any sections like "History", "See also", etc.

Some examples of map size inflation:
Paris - from 18MB to 65MB
Moscow - from 51MB to 92MB
Buenos-Aires - from 39MB to 70MB
Mexico City - 148MB to 180MB
(these numbers are outdated, but you get the picture)

The big point is:
Users download and keep map files for the sake of the map data, not for the auxiliary function of wikipedia offline reading!
Especially when they can't understand many languages wiki articles are stored in - it just becomes waste of space and bandwidth. IMHO compact map files is one of the notable advantages of OM.

So any significant map size increase should be considered very carefully weighing in the value most of the users will get from it.

I think some questions were not answered before rolling out of the wikipedia feature:

• what will be a total size increase if we include all OM languages? (now its limited to just 5)
• what will be a size increase for touristic centers (which are downloaded the most often) if we include all languages
• size increases if we include all languages but article summaries only?
• if we include english summaries only?

It might happen that sizes will be significantly inflated even if we limit to summaries only but include all OM languages.
I don't think it'll be fair to keep the feature limited to a few languages only like it is now - in this case users who don't understand these languages will be at big disadvantage (no value added for them, but need to cope with bigger files anyways) - and this is users from mostly third-world countries who don't posses modern devices with lots of storage and cheap traffic.

Originally posted by @pastk in #2410 (comment)

[pastk]

An even better solution in the long term would be to make wiki articles downloadable as optional separate files.

[biodranik]

Exactly. Downloading wiki articles separately, in only a needed language, will save us a lot of traffic on map updates, while keeping wiki articles as detailed as possible.

[biodranik]

1. What's the wiki section format in mwm?
2. Are articles embedded into features directly or are they stored separately?
3. How features and articles are connected?
4. How to disconnect them to download separately, but make it easier to highlight features with wiki articles e.g. using search?
5. How to use an article size to rank features?
6. What would be the best and convenient UX to download wiki articles separately and only in the needed language(s)?

[jumelles]

6. What would be the best and convenient UX to download wiki articles separately and only in the needed language(s)?

There will absolutely be places in non-English-speaking countries that have a Wikipedia article in that language but none in English. Ideally the OSM data will link to Wikipedia via Wikidata, which should be vastly more comprehensive than just the English Wikipedia article pagespace.

Here's a quick example I found: Église Saint-Jacques-le-Majeur d'Asquins

[biodranik]

@jumelles how is that relevant for a user if he/she doesn't read that local language and for our discussion? Each feature still has a wiki link to check it online if necessary.

[jumelles]

I'm just pointing out to be sure to query Wikidata instead of Wikipedia.

[biodranik]

@jumelles Wikidata should be already queried. Can you please check if it already works in Organic Maps? And let's focus on the issue's topic.

[newsch]

1. What's the wiki section format in mwm?
2. Are articles embedded into features directly or are they stored separately?
3. How features and articles are connected?

It's the descriptions section, briefly described in descriptions/serdes.hpp

The section stores mappings of feature_index -> (lang, description).

The contents are:

• "FeaturesIndices" sorted array of feature indices
• "LangMeta" unordered array of (language, description index)
• "LangMetaIndex" array of LangMeta offsets in same order as FeaturesIndices
• "Strings" block compressed descriptions

So to get the description for a feature in a given language:

1. Binary search for the feature index in FeaturesIndices.
2. Get LangMeta offset in LangMetaIndex w/ same offset as feature index.
3. Get LangMetaIndex entry with offset.
4. Find description index for language in entry.
5. Get description index from Strings.

4. How to disconnect them to download separately

In the current form, disconnecting is straightforward: move the section to a separate file. It is tied to the layout of features in the main file however, so they must be updated together.

4. ...make it easier to highlight features with wiki articles e.g. using search?
5. How to use an article size to rank features?

With the current format you'd need to do all of the same lookups to get article size. Detecting presence means searching only FeaturesIndices if you don't care about language, or LangMetaIndex and LangMeta if you do.

Does search build its own index for ranking, or does it use the files directly?

[biodranik]

There is a separate index for everything, including search )

[biodranik]

CC @newsch