This branch contains code fixes needed to get all 3 project data sources loaded into the database.
It also switches the manifest.csv to use the new S3 bucket from the Code for DC AWS account, instead of the one from Neal's personal account that we had been using.
…e unique_data_id results in an empty table, delete the table before loading the data. This allows for the addition/removal of columns in a single table, as the code will recreate the table after it is deleted, using the current version of meta.json Note this will not work for tables that have more than one unique_data_id in them, because the table will never be empty Also handles errors due to missing or not found data in the zone_facts table - if a data field is not found, we want to log the error but don't want to terminate the code, for when this is run on a server.
…pdating the database. This handles the situation where there are multiple unique_data_ids in a database table but we want to update the table structure without rebuilding the entire database.
…rently it uses a simple string-matching implementation; TODO is to add more advanced info Splits the DHCD data into 'project' and 'subsidy' tables per the same logic used for the DCHousing data set
… database, by querying the MAR API Refactors the methods off of the 'BaseApiConn' into a new 'ProjectBaseApiConn' so that the Project api connection can use the MarApiConn without causing circular import issues when the MarApiConn tries to load the BaseApiConn Changes both DHCD and DCHousing apiconn objects to use the `ProjectBaseApiConn`
…elds, renames Proj_zip field (changed case from Proj_Zip), updates manifest - Updates all files in manifest to use new S3 bucket on the codefor dc account: `housing-insights` (instead of without the dash, which was bucket name on Neal's personal S3 account) - Fixes project loading bug re: incomplete refactor of fields needed for rename-census-tracts - Fixes bugs related to new ProjectBaseApiConn - split into separate file to avoid more circular import issues.
… and projects tables due to missing if statement
…s we will want, but gets these fields usable for now.
jkwening left a comment •
Lines 259 - 263 in Cleaner.py was throwing a KeyError. I believe the issue is that lat/lon and x/yxcoord lookups return a json object that has 'Table1' as the first level key so replacing 'returnDataset' key value and removing 'sourceOperation' validation fixed the error. Regarding the latter - it looks like you're trying verify that if address lookup it actually returns an address but this will throw a 'KeyError' if lat/lon or x/ycoord lookup returns empty result so you will need an additional condition for that to work correctly.
Hmm, so it looks like
This also means that this line
For now I'll hotfix this by putting the