Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store additional geocoding results #4020

Merged
merged 6 commits into from
Jul 9, 2023
Merged

Conversation

nllong
Copy link
Member

@nllong nllong commented May 3, 2023

Any background context you want to provide?

During geocoding, there are several additional fields that are returned that can be useful such as neighborhood. These results are now stored in reserved SEED columns (e.g., Geocoded Neighborhood, Geocoded City, etc.).

What's this PR do?

  • Fix the pre-commit file to not throw a warning about regexes
  • When calling "geocode" persist off the additional data that is collected
  • get_or_create the additional columns as needed and prevents those columns from being mappable

How should this be manually tested?

I had to create a new MapQuest API key since their auth systems seems to have been upgraded. I did need to provide a credit card even for the free account. I used privacy.com with a new credit card set with $1/year limit... just in case.

  • Create new MapQuest API Key
  • Put key into the settings
  • Import some addresses with only address and state
  • Verify that the additional geocoded data are automatically created

What are the relevant tickets?

n/a

Screenshots (if appropriate)

@github-actions
Copy link

github-actions bot commented May 3, 2023

Label error. Requires at least 1 of: Feature, Bug, Enhancement, Maintenance, Documentation, Performance, Do not publish. Found:

@nllong nllong added the Feature Add this label to new features. This will be reflected in the change log when generated. label May 4, 2023
@nllong nllong force-pushed the add-more-geocoding-results branch from 6069110 to 93e5309 Compare May 4, 2023 05:05
@nllong nllong requested a review from axelstudios May 4, 2023 05:05
@kflemin kflemin self-requested a review June 15, 2023 22:00
@kflemin
Copy link
Contributor

kflemin commented Jun 21, 2023

@nllong verified as working! had to get off the VPN and refresh the view to "show only populated columns" but then I could see the new geocoded results. 👍👍

Screen Shot 2023-06-20 at 10 00 24 PM

@@ -198,7 +243,12 @@ def _address_geocoding_results(id_addresses, mapquest_api_key):

response = requests.get(request_url)
try:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing: the UI hangs if on VPN...is there a way to catch and handle the SSL error here?
from the log, error was:
File "/Users/kflemin/repos/seed-TESTING3/seed/data_importer/tasks.py", line 1379, in _geocode_properties_or_tax_lots
geocode_buildings(property_state_qs)
File "/Users/kflemin/repos/seed-TESTING3/seed/utils/geocode.py", line 91, in geocode_buildings
address_geocoding_results = _address_geocoding_results(id_addresses, mapquest_api_key)
File "/Users/kflemin/repos/seed-TESTING3/seed/utils/geocode.py", line 244, in _address_geocoding_results
response = requests.get(request_url)
File "/Users/kflemin/.pyenv/versions/newseedenv/lib/python3.9/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/Users/kflemin/.pyenv/versions/newseedenv/lib/python3.9/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/kflemin/.pyenv/versions/newseedenv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/Users/kflemin/.pyenv/versions/newseedenv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/Users/kflemin/.pyenv/versions/newseedenv/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.mapquestapi.com', port=443): Max retries exceeded with url: /geocoding/v1/batch?&inFormat=json&outFormat=json&thumbMaps=false&maxResults=2&json=%7B%22locations%22:%20%5B%7B%22street%22:%20%2215013%20Denver%20West%20Parkway,%20Colorado%22%7D,%20%7B%22street%22:%20%228604%20Sunset%20Blvd,%20California%22%7D,%20%7B%22street%22:%20%221222%20Hunstpoint%20Way,%20Kentucky%22%7D%5D%7D&key=XXXXXX (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)')))

# Conflicts:
#	.pre-commit-config.yaml
@axelstudios axelstudios merged commit c99e4ca into develop Jul 9, 2023
7 checks passed
@axelstudios axelstudios deleted the add-more-geocoding-results branch July 9, 2023 19:05
dhaley pushed a commit that referenced this pull request Jul 21, 2023
* save the more detailed geocoding results to the records

* formatting and readability

---------

Co-authored-by: Alex Swindler <alex.swindler@nrel.gov>
@RDmitchell
Copy link

@nllong -- I am curious why this feature was implemented ... I guess it's good for the user to see the values used for geocoding, but is there more to it than that?

Also, I didn't have to create a new Mapquest key .. ??

@nllong
Copy link
Member Author

nllong commented Aug 10, 2023

If a user doesn't have all the geocoded data, for example they are missing city, then the city will be returned and stored in the geocoded_city location. In brief, there is more data in the geocoding that the user might want to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Add this label to new features. This will be reflected in the change log when generated.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants