Repository navigation
Releases
Releases · dstoffels/localis
v3.0.0
Data
2026-11-01 08:11 UTC
| Dataset | Records | Added | Removed | Changed | Changed fields |
|---|---|---|---|---|---|
| Scripts | 226 → 226 | 0 | 0 | 1 | aliases (1) |
| Languages | 7,923 → 7,923 | 0 | 0 | 13 | scripts (12), aliases (1) |
| Countries | 281 → 281 | 0 | 0 | 5 | languages (3), groupings (2) |
| Subdivisions | 51,833 → 51,827 | 0 | 6 | 15 | aliases (12), name (4), admin_level (1), parent (1) |
| Cities | 236,652 → 236,866 | 226 | 12 | 252 | population (123), subdivisions (116), name (15), lat (5), lng (5) |
v3.0.0rc1
Changed
- The Test workflow no longer runs the data pipeline on pushes to
dev. It fetched every source as published that day, so it failed whenever GeoNames or Wikidata had changed since the last refresh, whether or not the push touched the pipeline; upstream changes reach the data through the monthly ingest workflow, and the pipeline's logic stays covered by its unit tests.
Data
2026-10-25 04:10 UTC
| Dataset | Records | Added | Removed | Changed | Changed fields |
|---|---|---|---|---|---|
| Subdivisions | 51,709 → 51,833 | 131 | 7 | 25 | aliases (25), name (15) |
| Cities | 236,353 → 236,652 | 303 | 4 | 1,401 | lng (1,249), lat (1,245), name (134), subdivisions (111), population (33) |
Records renamed, added and removed
2026-10-19 14:53 UTC
| Dataset | Records | Added | Removed | Changed | Changed fields |
|---|---|---|---|---|---|
| Subdivisions | 51,710 → 51,709 | 0 | 1 | 6 | aliases (6), name (1) |
| Cities | 236,041 → 236,353 | 319 | 7 | 79 | population (67), name (8), lat (7), lng (7), subdivisions (3) |
v3.0.0b2
Added
Country.continent, aContinentwithname,geonames_codeandgeonames_idfor the seven continents, from GeoNames, filterable withcountries.filter(continent=...)by name or GeoNames code;Nonefor a country GeoNames doesn't list. Continents are GeoNames enrichment, not an authoritative standard, and separate from macroregions (#25).- Each release's Data subsection: every data refresh it ships, dated, with each dataset's record counts and changed fields, linked to the records it renamed, added and removed in
DATA_CHANGELOG.md, recorded back to 2026-10-04.
Changed
- The installed package is about 40% smaller (about 32MB, from 53.6MB): every data file now ships gzipped. The data and the API are unchanged (#29).
- Each data file starts with a header naming its columns after the fields its registry serves, and loading one built for another version raises
ValueErrorwith both column lists instead of misreading its rows.
Fixed
cities.search()ranks cities with the same score by population, largest first, so "Paris" returns Paris, France before Paris, Ontario, and a large city among many same-name places is never cut before it's scored. Reported by @SchubmannM in #8 (#26).- The ingest pipeline's cldr-json commit lookup authenticates with the workflow's token, so a scheduled or CI run no longer fails on the rate limit GitHub-hosted runners share (#27).
- Subdivision skill decisions are rechecked on every ingest run instead of applying permanently: a merge whose GeoNames record was deleted or is named by another decision, or an add-as-is GeoNames now has a matching record for, goes back for review rather than shipping a duplicate or being replaced by an unreviewed fuzzy match, and a decision whose code ISO removed is retired with its history kept (#28).
Data
2026-10-11 23:52 UTC
| Dataset | Records | Added | Removed | Changed | Changed fields |
|---|---|---|---|---|---|
| Countries | 281 → 281 | 0 | 0 | 252 | continent (252) |
2026-10-11 21:33 UTC
| Dataset | Records | Added | Removed | Changed | Changed fields |
|---|---|---|---|---|---|
| Subdivisions | 51,711 → 51,710 | 1 | 2 | 409 | name (335), aliases (323) |
| Cities | 235,970 → 236,041 | 71 | 0 | 22 | subdivisions (13), population (6), lat (3), lng (3), name (2) |
v3.0.0b1
Upgrading from 2.1.0
- Entity collection fields (
aliases,scripts,macroregions,groupings,currencies,languages) are lists instead of tuples, likeCity.subdivisionsalready was. Compare them with lists, and usetuple(...)where you need one as a dict key or set member. Subdivision.typeandSubdivisionBase.typeareNoneinstead of""for the GeoNames-only subdivisions, which have no ISO type, like every other missing field. Checktype is None, and filter them withsubdivisions.filter(type=localis.MISSING).filter(field="")no longer matches records with no value in the field, which was an accident of the old filter index. Uselocalis.MISSING.filter()with no field raisesTypeErrorinstead of returning[]. Iterate the registry to get every record.search()raisesTypeErrorfor a keyword argument it doesn't take, instead of ignoring it. Remove the argument.lookup()raisesTypeErrorfor a key that isn't a string or an int, instead ofAttributeErrorforNoneor a float, and no longer takesTrueas the integer 1. Check forNonebefore looking a key up.len(countries)leaves out historic entries unless they're included, like iteration,filter()andsearch()(250 rather than 281). Callcountries.set_include_historic(True)for the count with them.- The registries' undocumented
count,build_cache()andinvalidate_cache()are removed. Uselen()forcount; the other two were internal. - A query in a non-Latin script no longer finds a subdivision through its aliases, since those aliases were removed (see Removed). Query in Latin script.
Added
localis.currencies: every ISO 4217 code (from iso-codes) withlookup()by alpha3 or numeric,filter()andsearch()by name, andCountry.currencies, each current country's legal tender per Unicode CLDR, filterable withcountries.filter(currency=...)by name or alpha3; historic countries have nonelocalis.languages: every ISO 639-3 code (from iso-codes) withlookup()by ISO 639-3, 639-1 or 639-2/B code,filter()by name,scope,typeorscript, andsearch(); Unicode CLDR's English language names are aliases andLanguage.scriptslists CLDR's scripts for each, with CLDR'ssecondaryflag.Country.languageslists each current country's languages with an official status per CLDR, oneCountryLanguageper language and script with itsstatus,population_percentandscript, filterable withcountries.filter(language=...)localis.scripts: every ISO 15924 code (from iso-codes) withlookup()by alpha4 or numeric, andfilter()andsearch()by name, Unicode CLDR's English script names included as aliaseskeyon every entity, nested ones included: the stable reference to store instead ofid, which a data release renumbers, resolved bylookup()in any later version (alpha2or a historic entry'salpha_4, a subdivision's ISO code or else its GeoNames code, a city's GeoNames ID, a macroregion's code). The README documents IDs as valid only within an installed versionlocalis.MISSINGfilters for records with no value in a field, whichNonecan't, since it means "don't filter on this field":subdivisions.filter(type=MISSING)for GeoNames-only subdivisions,cities.filter(subdivision=MISSING)for cities linked to no subdivision,countries.filter(macroregion=MISSING)for historic countries without onewikidata_changedorphans: a Wikidata crosswalk mapping that would relink a subdivision from what it previously resolved to goes to the resolve-subdivisions skill instead of applying silently- The data's licenses travel with it:
src/localis/data/NOTICEattributes each source,LICENSES/holds the LGPL-2.1-or-later, CC-BY-4.0, Unicode-3.0 and CC0-1.0 texts, and the package metadata declaresMIT AND LGPL-2.1-or-later AND CC-BY-4.0 AND Unicode-3.0 AND CC0-1.0instead ofMITalone, which covered only the code localis.__version__, and top-level exports ofEntity,Missingand the registry classes (Registry,QueryableRegistry,CountryRegistryand the rest) for annotating code that takes themdocs/versioning.md: the versioning policy, covering what the public API is, what each release can change, deprecation before removal, the beta and release-candidate stages, and which releases are supported- Package metadata: the
Typing :: Typedclassifier, currency, language and script keywords, a changelog link, and the homepage pointing at the repository instead of the PyPI page
Changed
- Historic countries take their Wikidata names as aliases by ISO 3166-3 alpha-4 code (P773) rather than by a former alpha-2 no other entry shares, so entries whose code ISO reused are named too: with
include_historicset, "Czechoslovakia", "Soviet Union" and "Yugoslavia" find their entries. 25 of the 31 historic entries now carry Wikidata aliases, up from 4 - Text normalization folds Latin only: accents are stripped and Latin letters without a Unicode decomposition (ß, æ, ø, ł, ı, ə and others) are spelled out, while other scripts are no longer transliterated
- A city whose GeoNames name is written in another script takes GeoNames' ASCII name (24 cities)
- Every entity a call returns is built fresh, its nested records and lists included, so it can be changed freely without affecting the loaded data or later results. Entities are hashable by record, so they can go in a set or be a dict key
- Country search matches alpha-3 codes, so
countries.search("USA")finds the United States, and Wikidata's English short names are kept even where they're also the country's IOC or FIFA code, adding UAE, RSA and GDR as aliases - Countries leave historic entries out of
filter()andsearch()before ranking rather than after, so a search no longer reads every country to size an over-fetch and a historic entry can't take a shortlist slot from a current one cities.set_population_threshold()raises when called with a value that isn't a non-negative int, instead of failing on the next cities access, and setting the threshold already in place keeps the loaded cache- Cities ship a short-name list like countries and subdivisions, so short city names reachable only by an exact query (such as "Jīān") are found through the short-query fallback, for about 474 KB
- The filter index ships inverted, like the search index: each field value's ids are packed at ingest and sliced out at load, instead of the index being rebuilt from per-record rows on every load, so filters' first use is faster, most of all for cities
- Building entities is faster: each result reads its row position once rather than on every field, so iterating every city takes about a fifth less time
- Languages' and subdivisions' loaded data is smaller: the few scope and type values those records repeat are stored once each, about 0.9 MB less for languages and 0.3 MB for subdivisions
- With a population threshold set, the cities indexes load faster: each entry is checked against the set of ids the filtered cache kept, instead of through a predicate call
- The README is organized by query API (
get,lookup,filter,search, iteration) with each registry's specifics under Entities, where each entity's fields are a table marking which ones its nested form keeps. methodology.md lists the pipeline's four merge stages in order and keeps its one-off validation checks, dated and with how each was determined, in Discovery - Generated doc figures are marked with
<stat key="source.key:format">tags instead of HTML comments, and more figures are generated: the crosswalk's size and disagreements, the non-administrative groupings, Kosovo's records and the cities named in ASCII - The data refresh commits the Wikidata results it built from, the subdivision crosswalk (
ingest/subdivisions/inputs/wikidata_crosswalk.json) and the country names (ingest/countries/inputs/wikidata_country_names.json) and accepts a removal of committed entries only once a repeat query is missing the same ones - The data refresh builds every dataset from its sources on every run and replaces the shipped data only when the whole run succeeds, including a final reconcile check of the new build's counts, so the shipped datasets always come from one run and a failed run can't leave one dataset pointing at another's ids. Each source file is recorded by URL and SHA-256 as well as ETag, and each refresh reports the records it added, removed and changed per dataset. The documented figures are regenerated only after the new data replaces the old, and each refresh logs its steps and how it ended to
ingest/pipeline.log: succeeded, stopped on subdivision orphans, or failed in a named step - The data refresh fetches iso-codes and CLDR at their main branch's current commit rather than by branch name, so each source manifest records the exact upstream commit a release was built from
- The data refresh stops, naming the conflict and how to resolve it, when a source contradicts itself rather than silently keeping one side: a code iso-codes lists twice, a code CLDR's territory containment makes two kinds of macroregion, a script CLDR gives a language as both primary and secondary, or a lookup key two records would share. A script CLDR repeats for one language ships once
- The data refresh retries failed downloads with backoff, like Wikidata queries, and treats a download cut off short of its length as failed rather than complete
- The data refresh downloads only the sources whose ETag changed, rather than every source of a stage when any one of them changed, so a CLDR update no longer downloads GeoNames' alternate names again, and a rerun after a failed or orphan-stopped refresh doesn't download again what that refresh already did
Removed
- The
unidecodedependency (GPL-2.0-or-later);rapidfuzzis now the only runtime dependency - Subdivision aliases written in non-Latin scripts (19,882), which only a query in that script c...
v2.1.0
v2.0.0
v1.1.2
What's Changed
- v1.1.2: index rework, entity/view/store refactor, dataset resourcing by @dstoffels in #11
Full Changelog: v1.1.1...v1.1.2
v1.1.1
v1.1.0
What's Changed
- Automated data ingest by @github-actions[bot] in #9
New Contributors
- @github-actions[bot] made their first contribution in #9
Full Changelog: v1.0.0a3...v1.1.0
v1.0.0a3
What's Changed
- BREAKING: sqlite deprecation by @dstoffels in #6
- Prerelease v1.0.0a3 by @dstoffels in #7
Full Changelog: v1.0.0a2...v1.0.0a3