Takes Waze CCP data feed and processes it into a cloud database for querying, analysis, API hooks, and mapping.
Louisville is creating an automated cloud processing solution that can be replicated by any CCP Partner, with the help of other govs, partners, and sponsors.
You grab this Terraform.io code and deploy the infrastructure stack (currently AWS but cloud agnostic).
You enter your CCP data feed URL as a parameter.
Then you can store, analyze, query, extract live and historic data for your city.
See the Projects area for how you can help, and the Wiki for all the details.
Organizations that are helping to fund, manage, promote, and support the project.
We have an end-to-end data processor and database working that you can deploy. It saves your CCP data as JSON files every 2 minutes, and processes the data into a combined real-time and historic database.
See /docs/technical-deployment.md for very detailed instructions!
This creates an infrastructure stack which has pings your custom Waze CCP data feed every 2 minutes and save the JSON to a new bucket, which then gets processed into the relational database. There is error handling and also notification options for when things go right or wrong.
Here's what was created:
You can update the stack with new infrastructure as the code here gets updated, and it only affects new and changed items. You can also remove all the infrastructure automatically (minus the S3 bucket you created manually) by deleting the Terraform stack using terraform destroy after the get and init commands.
You can also dump any previously collected historic JSON files into your bucket and the processor will go through them and save/update the relevant data into your database. Using aws s3 cp is a good place to start to copy files in chunks from a previous bucket to a new bucket.
See /docs/technical-backloading-data.md for very detailed instructions!
This config stands up infrastructure that is mostly cheap/free (depending on usage), but the database itself is pretty powerful and will result in monthly charges in excess of $200 (as of this writing). We are working on ways to reduce the costs and you can help out on this issue.
We are working on writing API hooks, data visualizations and tools, and maps, which is all part of our project roadmap.
See our Projects area for our blueprint of how we are proceeding.
We would like to collaborate with you! Please suggest updates, work on the help wanted issues, collaborate on the Wiki, etc. It would be great to work together to get the best solution, use cases, and finish faster.
We've build out the code in Terraform and supported AWS at first, but would like it to be deployed to any cloud provider. See our Issues area for how you can help with this.
If you'd like a little more background on Louisville and what our city has been going with Waze and other mobility data, take a look at these links: