Spin up a single node of Confluent Community components using Terraform.
A Terraform script to provision and configure an EC2 instance in AWS
A bash shell script to install and configure the following components:
- Centos7 node for Confluent
- Deploy the Snowflake Connector for Apache Kafka
- Create scripts to configure Kafka Connect connectors
- Create scripts to produce data in plain text and JSON using Python and Faker
Once deployed and configured, the Snowflake Connector for Apache Kafka will consume data from a Kafka topic and load the data into a table in a Snowflake account.
Currently, the only cloud provider supported is AWS.
The following architecture diagram depicts the components used in this example.
To run the examples, you will first need to install and configure, or have access, to the following software:
- An AWS account. Azure and GCP not yet supported.
- Terraform 0.12.3 or higher - installed and configured with AWS access keys.
- A Snowflake account - use recommended configuration for the Connector for Apache Kafka as described in the Snowflake documentation.
It is assumed that you are able to install and configure Terraform, configure credentials for AWS, and have a Snowflake account with the required privileges to perform the configuration of the Snowflake database and Snowflake Connector for Kafka.
You should create an IAM role for Terraform in the AWS Console and generate access keys for that role. i
Install the AWS CLI and configure it using these instructions. Terraform will then use the credentials configured for the CLI.
Install Terraform 0.12.3 or higher and configure it using these instructions.
You should have access to a Snowflake account and have the required privileges documented here.
You can get a Snowflake trial account here.
You will need to edit variables.tf located in the cloud provider subdirectory
Navigate to the cloud provider subdirectory and execute the following commands at the OS prompt:
terraform init
terraform planIf there are no errors, execute this command:
terraform applyEnter yes when prompted.
To remove the provisioned infrastructure, execute this command:
terraform destroy