Missing cities and countries

[pythobot]

Hello,

I have started trying out this library, but it seems to be missing cities and countries mostly from South America. What's the best way to update the cities.json and countries.json files? Is it ok just to add the data in there manually?

Also, how can this library map Shanghai as China, where is that relation mapped? why does it not behave the same for Caracas?

>>> geotext.extract(input_text="Living in Caracas", span_info=True)
{'cities': {'Caracas': {'count': 1, 'span_info': [(10, 17)]}}, 'countries': {}}

Thanks in advance!

[iwpnd]

Hi, thanks for using the library. :)

So, the demo data is super incomplete and only there to show the capabilities. You can however simply bring your own data as shown.

Also, how can this library map Shanghai as China, where is that relation mapped? why does it not behave the same for Caracas?

In the example, China is only found because it is specifically mentioned. Relations are not yet part of the library.

[nfrik]

I agree, unfortunately, the library doesn't recognize some countries/cities if typed with lower case:

geotext.extract(input_text='london is cool tashkent is not bad, but ukraine is not a city')

returns:

{'cities': {'London': {'count': 1, 'span_info': [(0, 6)]}}, 'countries': {}}

[iwpnd]

Hi @nfrik and thanks for your interest.

flashgeotext is case sensitive for two reasons.

1. Cities and countries are named entitities and therefor capitalized more often than not in sources such as newspaper articles, wikipedia articles, academic publications etcpp.
2. deliberately narrowing it down to a case sensitive lookup decreases the amount of false positive hits you get, when you want to also consider a bunch of synonyms for a city.

What I do agree with you on, is giving the user at least the option to turn off case sensitive lookup. This is a quick fix. I'm happy to receive a PR. Otherwise I work on it when I find the time to. :)

[Za-Re]

Hi @iwpnd

Enabling case sensitive option would be great. You're right about named entities in newspapers and articles. However another potential scenario could be extracting country names from social media posts and comments, e.g. in my case Tweets. They don't follow capitalization strictly. In this case, this option would be really helpful.

[iwpnd]

I know this is super old but I finally addressed it in v0.3.0 pending pip release.