Adding more languages to the JSON dumps — eight EU official languages are missing

[zdila]

Adding more languages to the JSON dumps — eight EU official languages are missing

Hi,

I maintain the geocoding setup for Freemap Slovakia (OZ Freemap Slovakia, the Slovak OpenStreetMap chapter), which runs freemap.sk and freemap.eu. We currently use nominatim.openstreetmap.org for search and are planning to move to a self-hosted Photon instance — exactly the migration the Nominatim usage policy asks users at our traffic level to make.

While preparing the import I ran into a blocker, and I wanted to ask about it before assuming it's intentional.

The gap

The language list published in the "A New Look for Photon Dumps" blog post is:

en, ru, zh, ja, uk, ar, ko, ca, fr, de, fi, be, pl, es, sr, br, he, sv, el, it, th, ga, oc, kn, ur, ms, nl, my, eu, ka, hu, fa, hi, pt, lt, ro, cs

I confirmed this against the actual data — the CountryInfo header of photon-dump-austria-1.0-latest.jsonl.zst carries name:cs but no name:sk.

Of the 24 EU official languages, eight are absent: Slovak, Slovenian, Croatian, Bulgarian, Danish, Estonian, Latvian, Maltese. Meanwhile the list includes Kannada, Burmese, Urdu, Thai, Breton and Occitan. I couldn't work out the selection principle, which is partly why I'm asking rather than assuming.

Why it matters

Because -languages can only select from what's in the dump, this isn't something a downstream user can configure around. Concretely: a Slovak user searching for "Paríž" (Paris) finds it on Nominatim today, because Nominatim indexes all name:* tags. With the Photon dumps it cannot work at all.

Slovak names inside Slovakia are fine via the local name tag — it's exonyms for foreign places that break. So for us the recommended migration away from the public Nominatim is a functional regression, and the same would apply to Croatian, Danish, Bulgarian and Slovenian users.

Why now

The 1.0.0 release notes state that "importing many languages into the database no longer slows down operations". If the historical reason for a curated list was import or query cost under the old ElasticSearch schema, that constraint appears to be gone.

There also seems to be precedent for growing the list:

• In how to get result based on my local language ? #360 (2018) a user reported that fa wasn't supported.
• In Support for custom language sets #494 (2020) another asked for Serbian — noting, as I am, that Nominatim had the name but without autocomplete.

Both languages are in the export set today. Discussion #528 mentions the dumps once carried only en/de/fr, so the list has clearly expanded over time.

(Related: #33, open since 2014, asks for the broader "index all name:*" case — I'm asking for something much narrower.)

Questions

1. What determines which languages are in the export set — is the selection documented anywhere?
2. Is there a specific reason sk was left out, or is it simply that nobody has asked?
3. Is adding a language purely a config change on the export server, or does it need changes in Photon itself?
4. What's the actual cost per additional language now, in dump size and import time? If the 1.0.0 note means it's cheap, would a broader default be on the table?
5. If this is worth pursuing, is a Discussion the right venue, or should it be an issue — and on this repo or the Nominatim side?

The ask

If the answers point that way, would you consider adding the missing EU official languages — or at minimum sk — to the export configuration? Happy to help test, or to run an import against a trial dump and report back.

Thanks a lot for the dumps and the export service generally — they're what makes self-hosting realistic for a small nonprofit like ours.

Martin Ždila
OZ Freemap Slovakia

[lonvia]

What determines which languages are in the export set — is the selection documented anywhere?

If I remember correctly, I simply looked at the most frequent name:* tags in OSM. Thinking about it now, I rather should have used statistics from Nominatim rather than taginfo as the former expands the name tag in single-language countries.

Is there a specific reason sk was left out, or is it simply that nobody has asked?

See above. I presume that there are simply not a lot of translations for names in Slovakian and name:sk won't be used inside Slovakia.

Is adding a language purely a config change on the export server, or does it need changes in Photon itself?

I can expand the list at any time on the export server. Reducing is more of an issue because people might already be using it and we don't want surprises for them. (I might make an exception for Breton because I just discovered that 80% of the names are made-up translations.)

What's the actual cost per additional language now, in dump size and import time? If the 1.0.0 note means it's cheap, would a broader default be on the table?

As long as we are talking about the JSON dump, I don't think there is any significant difference in size. I'm very much open to adding more languages. Can you give me the list of language codes for the languages above, so I don't have to look them up?

[zdila]

Thanks - that's a very useful answer, and the taginfo-vs-Nominatim point explains a lot. It would neatly account for why Slovak, Danish and Norwegian fell off while languages that get translated into a lot stayed on.

Here are the codes, grouped by priority so you can draw the line wherever makes sense.

Tier 1 - EU official languages currently missing (the eight from my first post)

sk Slovak · sl Slovenian · hr Croatian · bg Bulgarian · da Danish · et Estonian · lv Latvian · mt Maltese

Tier 2 - national languages of other European countries

no Norwegian (possibly also nb Bokmål / nn Nynorsk - OSM uses all three) · is Icelandic · sq Albanian · mk Macedonian · bs Bosnian · tr Turkish · lb Luxembourgish · fo Faroese · hy Armenian · az Azerbaijani

Norwegian stands out here: ~5M speakers, very actively mapped, and no dump language at all.

Tier 3 - regional/minority languages with real OSM usage

cy Welsh · gd Scottish Gaelic · gl Galician · fy West Frisian · rm Romansh · co Corsican · hsb/dsb Sorbian · fur Friulian · se Northern Sami · lld Ladin

This tier is where the quality concern you raised about Breton is most likely to bite, so it may be the natural place to stop - or to check counts first.

On picking the set generally

Rather than growing the list by request forever, would it be worth deriving it from Nominatim statistics with a frequency threshold, as you suggested? That would fix the taginfo bias at the root, be self-maintaining as OSM grows, and give a defensible answer when the next person asks for their language.

I understand additions are effectively one-way given you don't want to surprise existing users - so happy for you to start with Tier 1 only if that's the safer call. Since freemap.eu covers all of Europe and the UI is localised into several languages, though, the value for us really is the broader set rather than sk alone.

[lonvia]

I decided to go a different way. Nominatim has a list of official languages per country. Let's simply support all those. It is fair to assume that these represent languages in actual use as the list is somewhat curated. And it is easy to generate. The full list would be:

aa, af, am, ar, ast, ay, az, bal, be, ber, bg, bi, bn, bs, ca, crs, cs, da, de, dv, dz,
el, en, es, et, eu, fa, fi, fo, fr, ga, gl, gn, he, hi, ho, hr, ht, hu, hy, id, is, it, ja,
ka, kaa, kk, kl, km, ko, ku, ky, lb, lo, lt, lv, mfe, mg, mh, mi, mk, mn, ms, mt, my,
na, nb, nd, ne, niu, nl, nn, no, ny, om, pau, pih, pl, pnb, ps, pt, qu,
rar, rm, ro, ru, rw, sd, se, sf, sg, si, sk, sl, sm, sn, so, sov, sq, sr, srp, ss, st, sv, sw,
ta, tet, tg, th, ti, tk, tkl, tl, tn, tox, tpi, tr, uk, ur, uz, vi, xh, zgh, zh, zh-hans, zh-hant, zu

Adding br, kn, oc to that because it is already present.

[zdila]

That's a much better solution than a hand-maintained list — self-maintaining, and it gives a principled answer next time someone asks.

Checking it against what I sent: all of Tier 1 and all of Tier 2 are covered, including no/nb/nn, is, sq, mk, bs, tr, lb, fo, hy and az. That's everything we need, so from our side this fully resolves it. Thank you.

One observation, take it or leave it: the list picks up sub-national languages where a country's settings happen to list them — ca, eu and ast for Spain, rm for Switzerland, se for the Nordics — but misses ones whose countries only list the national language. The most notable is cy (Welsh), which has official status in Wales and is well represented in OSM via bilingual signage; gd (Scottish Gaelic) and fy (West Frisian) are similar cases. Not a blocker for us at all — just flagging in case you'd rather handle them the same way as br/kn/oc.

Also, sr and srp both appear, which I assume is a quirk of the source data rather than intentional.

Happy to run an import against the first dump that includes the new set and report back on size and import time.

[lonvia]

Good catch on 'srp', I've opened osm-search/Nominatim#4157

Sub-national languages remains a tricky topic and also something of a niche. Lets see if there is any demand.

I'm currently running tests on my own server and will deploy the list once I'm sure that the dump size doesn't explode.

[lonvia]

As expected, the compressed files practically doesn't change size and processing time is more or less the same. The new list is now deployed on the export server and should take effect next week.

[zdila]

Reporting back as promised.

Import: photon-dump-europe-1.0-latest.jsonl.zst, 2026-08-09 build (13,188,391,174 bytes), Photon 1.3.0, -languages sk,cs,en,de,fr,it,hu,pl,sl,hr,sr,uk,ro,nl,es,pt (16 languages).

Result (EPYC 9454P, NVMe, -Xmx16g, nice -n 10 alongside other production services):

• 158.1M documents in 6 h 45 min, ~6,500 docs/s average
• 59 GB final index on disk
• ~4,300 docs/s during the first hour, then steady at ~7,100 docs/s

The prebuilt photon-db-europe-1.0-latest.tar.bz2 is 30 GB compressed with the default four languages, but that isn't a like-for-like comparison and I have no control import to isolate the cost of the extra twelve. Happy to run a four-language Europe import on the same box if that number is useful to you.

The case from my first post works:

$ curl 'https://photon.freemap.sk/api?q=Paríž&lang=sk&limit=1'
"name":"Paríž", "state":"Île-de-France", "country":"Francúzsko", "countrycode":"FR"

Search on freemap.sk and freemap.eu now runs against this instance instead of nominatim.openstreetmap.org. Thanks for the quick turnaround.