Create ingest process for Chicago permits

[dfsnow]

Problem:

The CCAO needs a way to automatically ingest, transform, and clean Chicago's permit data in a format that can be exported to excel for review before it is uploaded to ias. Currently, this is all done manually.

Goal: a monthly process as described below

• Automatic ingest, data processing, cleaning, and storage in the data lake (specs tbd)
• Automatic surfacing of errors that can’t be automated away (like nonexistent PINs)
• Permits that have been amended are detected
• Additionally include property location address from ias for comparison

Collaborate with:

• Executive Assistant to Chief Valuations Officer

Documentation:

• In addition to well-documented code, please produce a 1-pager understandable to an executive without a coding background. It should clearly explain the problem, the method, the solution, the duration of the project, and the impact. Please have the Chief Data Officer review it so that it can be made public.

Upon completion, anticipate that the 1-pager will be sent by the project lead or by the CDO, ccing you, to:

• Chief Operations Officer
• Director of Data Integrity
• Chief Valuations Officer
• Chief of Staff

Desired additional benefits:

• More complete understanding of the permits table
• Gets us closer to thinking about how to use permits in sales val pipeline
• Gets us closer to publishing a permits OpenData asset

[dfsnow]

@jeancochrane This is basically done right? Can we close this?

[jeancochrane]

Yup, good flag @dfsnow! Code lives in https://github.com/ccao-data/extract-permits.