Repository navigation
v3.0.0b1
Pre-release
Pre-release
Upgrading from 2.1.0
- Entity collection fields (
aliases,scripts,macroregions,groupings,currencies,languages) are lists instead of tuples, likeCity.subdivisionsalready was. Compare them with lists, and usetuple(...)where you need one as a dict key or set member. Subdivision.typeandSubdivisionBase.typeareNoneinstead of""for the GeoNames-only subdivisions, which have no ISO type, like every other missing field. Checktype is None, and filter them withsubdivisions.filter(type=localis.MISSING).filter(field="")no longer matches records with no value in the field, which was an accident of the old filter index. Uselocalis.MISSING.filter()with no field raisesTypeErrorinstead of returning[]. Iterate the registry to get every record.search()raisesTypeErrorfor a keyword argument it doesn't take, instead of ignoring it. Remove the argument.lookup()raisesTypeErrorfor a key that isn't a string or an int, instead ofAttributeErrorforNoneor a float, and no longer takesTrueas the integer 1. Check forNonebefore looking a key up.len(countries)leaves out historic entries unless they're included, like iteration,filter()andsearch()(250 rather than 281). Callcountries.set_include_historic(True)for the count with them.- The registries' undocumented
count,build_cache()andinvalidate_cache()are removed. Uselen()forcount; the other two were internal. - A query in a non-Latin script no longer finds a subdivision through its aliases, since those aliases were removed (see Removed). Query in Latin script.
Added
localis.currencies: every ISO 4217 code (from iso-codes) withlookup()by alpha3 or numeric,filter()andsearch()by name, andCountry.currencies, each current country's legal tender per Unicode CLDR, filterable withcountries.filter(currency=...)by name or alpha3; historic countries have nonelocalis.languages: every ISO 639-3 code (from iso-codes) withlookup()by ISO 639-3, 639-1 or 639-2/B code,filter()by name,scope,typeorscript, andsearch(); Unicode CLDR's English language names are aliases andLanguage.scriptslists CLDR's scripts for each, with CLDR'ssecondaryflag.Country.languageslists each current country's languages with an official status per CLDR, oneCountryLanguageper language and script with itsstatus,population_percentandscript, filterable withcountries.filter(language=...)localis.scripts: every ISO 15924 code (from iso-codes) withlookup()by alpha4 or numeric, andfilter()andsearch()by name, Unicode CLDR's English script names included as aliaseskeyon every entity, nested ones included: the stable reference to store instead ofid, which a data release renumbers, resolved bylookup()in any later version (alpha2or a historic entry'salpha_4, a subdivision's ISO code or else its GeoNames code, a city's GeoNames ID, a macroregion's code). The README documents IDs as valid only within an installed versionlocalis.MISSINGfilters for records with no value in a field, whichNonecan't, since it means "don't filter on this field":subdivisions.filter(type=MISSING)for GeoNames-only subdivisions,cities.filter(subdivision=MISSING)for cities linked to no subdivision,countries.filter(macroregion=MISSING)for historic countries without onewikidata_changedorphans: a Wikidata crosswalk mapping that would relink a subdivision from what it previously resolved to goes to the resolve-subdivisions skill instead of applying silently- The data's licenses travel with it:
src/localis/data/NOTICEattributes each source,LICENSES/holds the LGPL-2.1-or-later, CC-BY-4.0, Unicode-3.0 and CC0-1.0 texts, and the package metadata declaresMIT AND LGPL-2.1-or-later AND CC-BY-4.0 AND Unicode-3.0 AND CC0-1.0instead ofMITalone, which covered only the code localis.__version__, and top-level exports ofEntity,Missingand the registry classes (Registry,QueryableRegistry,CountryRegistryand the rest) for annotating code that takes themdocs/versioning.md: the versioning policy, covering what the public API is, what each release can change, deprecation before removal, the beta and release-candidate stages, and which releases are supported- Package metadata: the
Typing :: Typedclassifier, currency, language and script keywords, a changelog link, and the homepage pointing at the repository instead of the PyPI page
Changed
- Historic countries take their Wikidata names as aliases by ISO 3166-3 alpha-4 code (P773) rather than by a former alpha-2 no other entry shares, so entries whose code ISO reused are named too: with
include_historicset, "Czechoslovakia", "Soviet Union" and "Yugoslavia" find their entries. 25 of the 31 historic entries now carry Wikidata aliases, up from 4 - Text normalization folds Latin only: accents are stripped and Latin letters without a Unicode decomposition (ß, æ, ø, ł, ı, ə and others) are spelled out, while other scripts are no longer transliterated
- A city whose GeoNames name is written in another script takes GeoNames' ASCII name (24 cities)
- Every entity a call returns is built fresh, its nested records and lists included, so it can be changed freely without affecting the loaded data or later results. Entities are hashable by record, so they can go in a set or be a dict key
- Country search matches alpha-3 codes, so
countries.search("USA")finds the United States, and Wikidata's English short names are kept even where they're also the country's IOC or FIFA code, adding UAE, RSA and GDR as aliases - Countries leave historic entries out of
filter()andsearch()before ranking rather than after, so a search no longer reads every country to size an over-fetch and a historic entry can't take a shortlist slot from a current one cities.set_population_threshold()raises when called with a value that isn't a non-negative int, instead of failing on the next cities access, and setting the threshold already in place keeps the loaded cache- Cities ship a short-name list like countries and subdivisions, so short city names reachable only by an exact query (such as "Jīān") are found through the short-query fallback, for about 474 KB
- The filter index ships inverted, like the search index: each field value's ids are packed at ingest and sliced out at load, instead of the index being rebuilt from per-record rows on every load, so filters' first use is faster, most of all for cities
- Building entities is faster: each result reads its row position once rather than on every field, so iterating every city takes about a fifth less time
- Languages' and subdivisions' loaded data is smaller: the few scope and type values those records repeat are stored once each, about 0.9 MB less for languages and 0.3 MB for subdivisions
- With a population threshold set, the cities indexes load faster: each entry is checked against the set of ids the filtered cache kept, instead of through a predicate call
- The README is organized by query API (
get,lookup,filter,search, iteration) with each registry's specifics under Entities, where each entity's fields are a table marking which ones its nested form keeps. methodology.md lists the pipeline's four merge stages in order and keeps its one-off validation checks, dated and with how each was determined, in Discovery - Generated doc figures are marked with
<stat key="source.key:format">tags instead of HTML comments, and more figures are generated: the crosswalk's size and disagreements, the non-administrative groupings, Kosovo's records and the cities named in ASCII - The data refresh commits the Wikidata results it built from, the subdivision crosswalk (
ingest/subdivisions/inputs/wikidata_crosswalk.json) and the country names (ingest/countries/inputs/wikidata_country_names.json) and accepts a removal of committed entries only once a repeat query is missing the same ones - The data refresh builds every dataset from its sources on every run and replaces the shipped data only when the whole run succeeds, including a final reconcile check of the new build's counts, so the shipped datasets always come from one run and a failed run can't leave one dataset pointing at another's ids. Each source file is recorded by URL and SHA-256 as well as ETag, and each refresh reports the records it added, removed and changed per dataset. The documented figures are regenerated only after the new data replaces the old, and each refresh logs its steps and how it ended to
ingest/pipeline.log: succeeded, stopped on subdivision orphans, or failed in a named step - The data refresh fetches iso-codes and CLDR at their main branch's current commit rather than by branch name, so each source manifest records the exact upstream commit a release was built from
- The data refresh stops, naming the conflict and how to resolve it, when a source contradicts itself rather than silently keeping one side: a code iso-codes lists twice, a code CLDR's territory containment makes two kinds of macroregion, a script CLDR gives a language as both primary and secondary, or a lookup key two records would share. A script CLDR repeats for one language ships once
- The data refresh retries failed downloads with backoff, like Wikidata queries, and treats a download cut off short of its length as failed rather than complete
- The data refresh downloads only the sources whose ETag changed, rather than every source of a stage when any one of them changed, so a CLDR update no longer downloads GeoNames' alternate names again, and a rerun after a failed or orphan-stopped refresh doesn't download again what that refresh already did
Removed
- The
unidecodedependency (GPL-2.0-or-later);rapidfuzzis now the only runtime dependency - Subdivision aliases written in non-Latin scripts (19,882), which only a query in that script could find: subdivisions' shipped data shrinks from 11,127 to 10,168 KB and their fully loaded memory from 49.7 to 43.0 MB. 274 Latin aliases that deduplication had previously dropped in favor of a Cyrillic spelling ship again
Fixed
- 448 GeoNames-only subdivisions carried an alias that was their own name in another case or spelling (Posavski Kanton's "Posavski kanton"), since only a subdivision merged with ISO had its aliases deduplicated against its name; every subdivision's now are
subdivisions.filter(country=...)matched a country's numeric code only unpadded ("76"or76); ISO's three-digit form ("076") now matches too- Generated sizes in the README and docs were computed in KiB and MiB but labeled KB and MB, understating every figure by 2.4% (KB) or 4.9% (MB); they're now computed in decimal KB and MB
- Search benchmarks counted a result as a miss when it was a different record with the same name as the sampled one (two "Franklin County" subdivisions), which no search can tell apart; such a result now counts as a hit, and the benchmarks report how many hits came that way
- The README's search benchmarks claimed 5,000 queries per registry; each registry's actual query count is now shown, and the host line names the machine the benchmarks ran on rather than the footprint's
- Type checkers flag a field a registry's
filter()doesn't have, since its signature no longer takes**kwargs - Type checkers no longer flag
from localis import countries(or any other public name) as a private import: the package declares its public names in__all__ lookup()resolves a numeric code or GeoNames ID given as a string (countries.lookup("840"),cities.lookup("5128581")), not only as an intsearch()honors alimitabove 200 instead of capping results therefilter()andsearch()raiseValueErrorfor alimitbelow 1, which a negative value had silently cut results byget(True)returned the record with ID 1, sinceTruecounts as the integer 1; a bool is no longer taken as an ID- Subdivision automerge could leave out an ISO subdivision that qualified against a name another two subdivisions contested, without merging or orphaning it, so it never shipped; none was affected, and every ISO subdivision is now checked to ship or await review
- A city whose name contains a double quote (Poselok Turisticheskogo pansionata "Klyazminskoe vodohranilische") had its name returned wrapped in extra quotes, with each inner quote doubled
- Filter and search indexes load correctly on big-endian machines, where their packed ids had been read in the wrong byte order
- Capital schwa (Ə) and reversed E (Ǝ) normalized to "@" and "3"; both now fold to "a", like their lowercase forms
- Subdivision automerge never stripped type words spelled with diacritics (járás, huyện, ilçesi, shahrestān) from names before matching, since names are compared accent-free; the type words are now folded the same way, and an unaccented spelling such as "Huyen" also marks a name's type
set_population_threshold()changes the filter under the registry's lock, so a thread loading cities at the same time can't build the cache and indexes from different thresholds- The data refresh records the ETag of the file it actually downloaded, read from the download itself rather than from a second request that could see a newer version
Security
- CI jobs that install or run project and third-party code no longer hold write tokens. Test is read-only, and its ingest check on
devfails instead of committing;ingest.yamlbuilds read-only and pushes its commits from a job running only git andgh;release.yamlbuilds read-only, publishes to PyPI from a job holding onlyid-token: write, then creates the GitHub release from one holding onlycontents: write - Every workflow action is pinned to a full commit SHA, with Dependabot proposing updates after a 7-day cooldown; a job's outputs reach later scripts as environment variables rather than pasted into them, the release jobs no longer restore uv's cache, and the GitHub release is created with
ghinstead of a third-party action - Release runs on push to
maininstead of onworkflow_run, a trigger that runs with the default branch's privileges whatever started it, fork PRs included;mainis no longer tested again after a merge, since its ruleset admits only PRs whose tests passed against an up-to-datemain - The release build is reproducible: CI installs a pinned uv verified by checksum, and
uv buildtakes hatchling and its dependencies only from hash-pinned build constraints; release tags can no longer be moved or deleted - SECURITY.md describes how to report a vulnerability privately, through the repository's Security tab
- Each release installs its built wheel in a fresh environment and checks it before publishing to PyPI, refuses a version without a CHANGELOG section, and takes its GitHub release notes from that section; Dependabot also proposes Python dependency updates, and CI tests every supported Python, 3.11 to 3.14