Plan to improve search on world capitals

[ingenieroariel]

For the Think Hazard project, we are going to make sure world capitals from OSM can be 100% found. For this, we will use a combination of the GAUL dataset to replace the older admin boundaries and will try make use of geonames.org geocoder to help improve the performance of the search.

Improving search has to be done in a way that we don't just fix what we notice is broken. For this, we will implement cross-validation and split the test set 50/50.

• Create a script to download the osm capitals and put it in capitals.json
• Extract the english name and sort it alphabetically in capitals.txt
• Create a curated list of capitals from Wikipedia
• Compare the two capitals list and decide which one to use for testing (We will use the one from WIkipedia, it has more entries).
• Split capitals list in test and validation
• Measure the performance of the current ThinkHazard prototype with the test set (41% of 131 cities were not found).
• Pair geonames records with the current GAUL records to be able to use geonames.org search and a reverse match via the database.
• Measure the performance of the new prototype with the test set
• Tweak the prototype to improve the performance with the test set
• Measure the performance of the current ThinkHazard prototype with the validation test
• Measure the performance of the new prototype with the validation test
• Share the new prototype with the ThinkHazard team based on the validation results

[ingenieroariel]

Here is the list I got from OSM (173 entries):

Abu Dhabi
Accra
Addis Abeba
Amman
Amsterdam
Ankara
Antananarivo
Apia
Ashgabat
Asmara
Astana
Asuncion
Athens
Baghdad
Baku
Bamako
Bandar Seri Begawan
Bangkok
Bangui
Banjul
Beirut
Belgrade
Belmopan
Berlin
Bern
Bishkek
Bissau
Bogota
Bratislava
Brazzaville
Bridgetown
Brussels
Bucharest
Budapest
Bujumbura
Cairo
Canberra
Cape Town
Chisinau
Colombo
Conakry
Copenhagen
Dakar
Damascus
Dili
Djibouti
Dodoma
Doha
Dublin
Dushanbe
Freetown
Gaborone
Georgetown
George Town
Guatemala City
Hanoi
Harare
Havana
Helsinki
Honiara
Islamabad
Jerusalem
Juba
Kabul
Kampala
Kathmandu
Khartoum
Kigali
Kingston
Kinshasa
Kuala Lumpur
Kuwait City
Kyiv
Libreville
Lilongwe
Lima
Lisbon
Ljubljana
Lome
London
Luanda
Lusaka
Luxembourg
Madrid
Majuro
Malabo
Managua
Manama
Manila
Maputo
Maseru
Mbabane
Minsk
Mogadishu
Monaco
Monrovia
Moroni
Moscow
Muscat
Nairobi
Nassau
Naypyidaw
N'Djamena
New Delhi
Niamey
Nicosia
Nouakchott
Nuuk
Oslo
Ottawa
Ouagadougou
Panama
Paramaribo
Paris
Phnom Penh
Podgorica
Port-au-Prince
Port Louis
Port Moresby
Port of Spain
Port Vila
Prague
Praia
Pretoria
Prishtina
Pyongyang
Quito
Rabat
Reykjavik
Riga
Riyadh
Rome
Sana'a
San Jose
San Juan
San Salvador
Santiago
Santo Domingo
Sao Tome
Sarajevo
Seoul
Singapore
Skopje
Sofia
St. George's
Stockholm
Sucre
Suva
Taipei City
Tallinn
Tashkent
Tbilisi
Tegucigalpa
Tehran
Thimphu
Tirana
Tiraspol
Tokyo
Tripoli
Tunis
Ulaanbaatar
Valletta
Vienna
Vientiane
Vilnius
Warsaw
Washington
Wellington
Windhoek
Yamoussoukro
Yaounde
Yerevan
Zagreb

[ingenieroariel]

Here is the list I got from Wikipedia 262 entries (using both official and defacto capitals):

Abu Dhabi
Abuja
Accra
Adamstown
Addis Ababa
Algiers
Alofi
Amman
Amsterdam
Andorra la Vella
Ankara
Antananarivo
Apia
Ashgabat
Asmara
Astana
Asuncion
Athens
Avarua
Baghdad
Baku
Bamako
Bandar Seri Begawan
Bangkok
Bangui
Banjul
Basse-Terre
Basseterre
Beijing
Beirut
Belgrade
Belmopan
Berlin
Bern
Bishkek
Bissau
Bogotá
Brasília
Bratislava
Brazzaville
Bridgetown
Brussels
Bucharest
Budapest
Buenos Aires
Bujumbura
Cairo
Canberra
Caracas
Castries
Cayenne
Charlotte Amalie
Chisinau
Cockburn Town
Conakry
Copenhagen
Dakar
Damascus
Dhaka
Dili
Djibouti
Dodoma
Doha
Douglas
Dublin
Dushanbe
East Jerusalem
Ramallah
Edinburgh of the Seven Seas
El Aaiún
Tifariti
Episkopi Cantonment
Flying Fish Cove
Fort-de-France
Freetown
Funafuti
Gaborone
George Town
Georgetown
Georgetown
Gibraltar
Guatemala City
Gustavia
Hagåtña
Hamilton
Hanga Roa
Hanoi
Harare
Hargeisa
Havana
Helsinki
Hong Kong
Honiara
Islamabad
Jakarta
Jamestown
Jerusalem
Juba
Kabul
Kampala
Kathmandu
Khartoum
Kiev
Kigali
King Edward Point
Kingston
Kingston
Kingstown
Kinshasa
Kuala Lumpur
Putrajaya
Kuwait City
Libreville
Lilongwe
Lima
Lisbon
Ljubljana
Lomé
London
Luanda
Lusaka
Luxembourg
Madrid
Majuro
Malabo
Malé
Mamoudzou
Managua
Manama
Manila
Maputo
Marigot
Maseru
Mata-Utu
Mbabane
Lobamba
Mexico City
Minsk
Mogadishu
Monaco
Monrovia
Montevideo
Moroni
Moscow
Muscat
Nairobi
Nassau
Naypyidaw
N'Djamena
New Delhi
Ngerulmud
Niamey
Nicosia
Nicosia
Nouakchott
Nouméa
Nukuʻalofa
Nuuk
Oranjestad
Oslo
Ottawa
Ouagadougou
Pago Pago
Palikir
Panama City
Papeete
Paramaribo
Paris
Philipsburg
Phnom Penh
Plymouth
Brades Estate
Podgorica
Cetinje
Port Louis
Port Moresby
Port Vila
Port-au-Prince
Port of Spain
Porto-Novo
Cotonou
Prague
Praia
Pretoria
Bloemfontein
Cape Town
Pristina
Pyongyang
Quito
Rabat
Reykjavík
Riga
Riyadh
Road Town
Rome
Roseau
Saint-Denis
Saipan
San José
San Juan
San Marino
San Salvador
Sana'a
Santiago
Santo Domingo
São Tomé
Sarajevo
Seoul
Singapore
Skopje
Sofia
Sri Jayawardenepura Kotte
Colombo 
St. George's
St. Helier
St. John's
St. Peter Port
St. Pierre
Stanley
Stepanakert
Stockholm
Sucre
 La Paz
Sukhumi
Suva
Taipei
Tallinn
Tarawa Atoll
Tashkent
Tbilisi
Kutaisi
Tegucigalpa
Tehran
Thimphu
Tirana
Tiraspol
Tokyo
Tórshavn
Tripoli
Tskhinvali
Tunis
Ulaanbaatar
Vaduz
Valletta
The Valley
Vatican City
Victoria
Vienna
Vientiane
Vilnius
Warsaw
Washington, D.C.
Wellington
West Island
Willemstad
Windhoek
Yamoussoukro
Abidjan
Yaoundé
Yaren
Yerevan
Zagreb