Example Product/User Analytics System Using Apache Kafka, AWS Redshift, and Metabase
This is an example of a system that captures a large stream of product usage data, or events, and provides both real-time data visualization and SQL-based data analytics. The stream of events is captured by Apache Kafka and made available to other downstream consumers. In this example, there are two downstream consumers of the data. The data flowing through Kafka can be viewed in near real-time using a web-based data visualization app. The other consumer stores all the data in AWS Redshift, a relational database that Amazon describes as "a fast, scalable data warehouse." Then we can query and visualize the data in Redshift from a SQL-compliant analytics tool. This example uses Metabase deployed to Heroku. Metabase is an open-source analytics tool used by many organizations, large and small.
This entire system can be deployed in 15 minutes -- most of that time spent waiting for Heroku and AWS to provision services -- and it requires very little ongoing operational maintenance.
Here's an overview of how the system works.
This project includes 3 apps:
- A data producer called
generate_data. Data is simulated in this example, but this could be replaced with almost anything that produces data: a marketing website, a SaaS product, a point-of-sale device, a kiosk, internet-connected thermostat or car. And more than one data producer can be added.
- A real-time data visualizer called
viz, which shows relative volume of different categories of data being written into Kafka.
- And a Kafka-to-Redshift writer called
reshift_batch, which simply reads data from Kafka and writes it to Redshift.
They all share data using Apache Kafka on Heroku.
You can optionally deploy Metabase to Heroku to query Redshift. Check out Metabase's Heroku Deploy Button.
- An AWS Redshift cluster. Check out this Terraform script for an easy way to create a Redshift cluster along with a Heroku Private Space and a private peering connection between the Heroku Private Space and the Redshift's AWS VPC. Not free! This will incur cost on AWS and Heroku.
Deploy to Heroku
git clone email@example.com:heroku-examples/kafka-stream-viz.git cd kafka-stream-viz heroku create heroku addons:create heroku-kafka:basic-0 heroku kafka:topics:create ecommerce-logs heroku kafka:consumer-groups:create redshift-batch heroku config:set KAFKA_TOPIC=ecommerce-logs heroku config:set KAFKA_CONSUMER_GROUP=redshift-batch heroku config:set FIXTURE_DATA_S3='s3://aws-heroku-integration-demo/fixture.csv' git push heroku master
Alternatively, you can use the Heroku Deploy button:
And then create the necessary Kafka topic and consumer group:
heroku kafka:topics:create ecommerce-logs #this can also be created at https://data.heroku.com/ heroku kafka:consumer-groups:create redshift-batch
Optionally, you can deploy Metabase to Heroku and use SQL to query and visualize data in Redshift. Use Metabase's Heroku Deploy button. Once deployed, you'll need to configure Metabase with the Redshift cluster URL, database name, username, and password.
git clone firstname.lastname@example.org:heroku-examples/kafka-stream-viz.git npm i
The following environment variables must be defined. If you used the Heroku deploy instructions above, all of the variables are already defined except for
DATABASE_URL: Connection string to an AWS Redshift cluster
FIXTURE_DATA_S3: S3 path to CSV of fixture data to load into Redshift before starting data stream through Kafka (e.g. s3://aws-heroku-integration-demo/fixture.csv)
KAFKA_URL: Comma-separated list of Apache Kafka broker URLs
KAFKA_CLIENT_CERT: Contents of the client certificate (in PEM format) to authenticate clients against the broker
KAFKA_CLIENT_CERT_KEY: Contents of the client certificate key (in PEM format) to authenticate clients against the broker
KAFKA_TOPIC: Kafka topic the system will produce to and consume from
KAFKA_CONSUMER_GROUP: Kafka consumer group name that is used by
redshift_batchprocess type to write to Redshift.
KAFKA_PREFIX: (optional) This is only used by Heroku's multi-tenant Apache Kafka plans (i.e.
Then in each of the
redshift_batch directories, run
Open the URL in the startup output of the
viz app. It will likely be