-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Geoparsing don't match some city #6
Comments
geoparsepy will by default remove location names that match common words in WordNet, name lists and stop lists. The whitelist allows you to prevent this removal for specific words you know are OK. this ensures they are in the location cache available for matching. I suggest you check the location cache to see if it contains 'sciacca' (without the whitelist entry). if the name is missing then its triggering the geo_parse_lib.is_good_place_name() method. this means it might appear on a blacklist, names list of stopword list maybe and is rejected as a 'bad name'. if the name is in the location cache OK but still does not match, then its being rejected as a viable location match. maybe the token 'sciacca' is being aggregated with other words (e.g. street prefixes) and so removed from the matching pool. try a sent with one word 'sciacca' to check if the word without context matches. |
The position "sciacca" is present in cached_locations, but is not present in indexed_locations (and therefore not matched) This is the cached_locations situation:
This is the indexed_locations situation:
To solve the problem without using the withelist I modified the geoparsepy.geo_parse_lib.is_good_place_name method with this code (I overwrote it):
This is the indexed_locations situation after modification:
A future upgrade for this library could be to insert a blacklist of rubbish OSM names for each language so as not to exclude most of the small cities that do not pass the previous tests. |
Thanks CalogeroCantone for your workaround. there is a tradeoff between compiling blacklists (which would contain a lot of names) and the heuristic filters in geoparsepy.geo_parse_lib.is_good_place_name() method. I will consider adding a config hyperparam to allow the phrase length filter to be configured, then users can change that setting to reflect thier appitite for false matches. The reported problem was actually because you had two languages specified ['it','en']. This means nltk English first names are loaded, and an English name is 'Di' which is short for 'Diana'. This was matched to your sentence with 'di Sciacca' and thus rejected as a probable full name (as 'Di London' would be). I tested with ['it'] and Sciacca was matched OK. My advice is to load only the language code of the text you will geoparse, and be careful when trying multiple languages. You can (if you have enough memory) load two instances of cached_locations() using two seperate configs, one for each language. This will produce a more precise output. |
I tried to geoparse some phrases but not all the city are matched (for example: 'Sciacca' and 'Asciano').
Note that all the city are present on the database and all the phrases are correctly tokenized.
EDIT: I noticed that if I manually whitelist the cities everything works fine, but why are they not shown directly?
Here is my code:
The output is the following:
The text was updated successfully, but these errors were encountered: