SOS DATA

Tested on Mac OS Catalina

This code is to build a postgreSQL database that houses Travis County Appraisal District records and Secretary of State Corporate filings.

Dependancies Setup

• Follow the rules here to install postal then follow the rules here to download the postgreSQL extension or comment out the call to run_schema.py for address normalization in build_database.py if you do not want address normalization in the sql database
• Create a virtual python environment and pip install -r requirements.txt

Database Setup and Population

• Use postgreSQL to create a database named Sos_data_fun
• Create a .env file with LOCAL_DATABASE_URL set to your database url and LOCAL_DEV="true"
  • Use psql \conninfo to get info to make url postgresql://[user[:password]@][netloc][:port][,...][/dbname]
• Create a folder named "data" in the projects parent directory and place your fixed width txt files extracted from the SOS bulk file export there
• Create a subfolder in "data" named "TCAD"
  • Comment out the call to tcad.read_tcad() and uncomment the call to tcad.download_read() if you have not already downloaded the [TCAD] data and populated the folder with it
• Run build_database.py once the above steps are completed to populate and build the SQL tables

Data Cleanup

If the database needs to be re made take the following steps:

• Remove the error.log file
• Run dropdb Sos_data_fun
• Run createdb Sos_data_fun
• Remove the .sql files in ./sql EXCEPT create_normalized_addresses.sql and create_name_search_index.sql
• Run build_database.py once the above steps are completed to populate and build the SQL tables

Contribution Notes

• Try to keep everything consistant with PEP 8
• Please document functions with docstrings
• Please file issues if anything is broken or incorrectly specified in the README
• Please try to keep consistant with the current module organization

To Do

• Build search indexes for names in psql
• Link up meta data tables (tables prefixed with md_)
• Record linkage with TCAD prop_owner_names and SOS names
• Record linkage with TCAD mailing_address and SOS addresses
• Write update_database() in clean_up.py
• Write tests for SQL data... gotta figure this out...