WIP: Make proj-addre one-to-many data and load into database. #497
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… the file from disk, but in the future this can be the class that checks for newly uploaded data in an online folder that admins can put data into - Adds a function to the base_project.py apiconn that uses the split address function to create a csv file of the one-to-many project to addresses. - Moves the split address function out of ingestion and into tools, since it's not used only in ingestion - Fixes bug in the MarApiConn - adds a method to query api using address string instead of generic string. Helps eliminate false positives. - Changes references to find_location to instead be find_addr_string to eliminate false positives TODO - Starts to add the one-to-many mapping to the create_project_subsidy_csv method for the dhcd and dchousing data sets. This is not complete so it instead breaks this code. Need to load the newly created prescat_addr.csv file into the database first and then resume coding this method.
NealHumphrey
changed the title
- in the section of the DHCD and DCHousing apiconn that makes the _subsidy and _project tables, it also makes an _addre table - Entity resolution in that section that checks for an existing NLHIC_id in the database now uses the _addre table, checking first for an exact address match, then looking for a fuzzy match for that address in the mar, and looking for an exact mar_id match; if neither address or mar_id matches, it assumes it's new. If any of the addresses match any of the addresses in the existing database table, it assumes the whole project matches - Adds the optional parameters to all of the ApiConn objects so that we can consistently pass database_choice to __init__, whether or not it is used - Small fix to how address splitting occurs to solve error - @jkwening would be good to check logic.
|
Updated description to reflect progress w/ most recent commit |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an "apiconn" object for the prescat. Currently this just loads the file from disk, but in the future this can be the class that checks for newly uploaded data in an online folder that admins can put data into
Adds a function to the base_project.py apiconn that uses the split address function to create a csv file of the one-to-many project to addresses.
Moves the split address function out of ingestion and into tools, since it's not used only in ingestion
Fixes bug in the MarApiConn - adds a method to query api using address string instead of generic string. Helps eliminate false positives.
Changes references to find_location to instead be find_addr_string to eliminate false positives
Fixes bug in address splitting code that found
andinMarlandorRandolph. Changed split string to' and '.Adds proj_addre table to database
DHCD and DCHousing ApiConn methods also create
_addre.csvfiles for uploadEntity resolution now done via the one to many mapping of addresses - if any of the addresses match, assumes the whole project matches.
To test
python/housinginsights/sources/folder runpython prescat.py(needs to be added to the options list in get_api_data.py, but should be ready to work there)/python/scripts:python load_data.py docker --update-only proj_addrepython get_api_data.pywith the code options edited to request theDCHousinganddhcdmodules.