kinesis-to-aurora

This is an Idempotent Scala service that persists user-events from kinesis stream to Aurora RDS.

Purpose

The main purpose of the project is to demonstrate

• http4s
• kamon
• AWS kinesis & KCL
• scalaz streams & Task implementation for AWS Kinesis stream
• scala & docker
• doobie
• circe
• statsd

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

What things you need to install the software and how to install them

Assumption

• you have
  1. [docker-id]
  2. [AWS account]

Installing

• set up development environment

  • scala 2.11+
  • sbt
  • docker
  • docker-compose
  • [JDK 8+]
  • direnv
  • *aws-cli*
  • *emacs ensime*

• set up environmnet variables

export DOCKER_USER='docker_userid'
export DOCKER_PASS='docker-password'
export AWS_REGION='us-east-1'
export AWS_ACCESS_KEY_ID=XXXXXX
export SECRET_ACCESS_KEY=XXXXXX
export JDBC_USER=xxxxx
export JDBC_PASSWORD=xxxxx
export JDBC_DB=mytestdb
export JDBC_URL='mysql://$JDBC_USER:$JDBC_PASSWORD@0.0.0.0:3306/$JDBC_DB'

Setup Project

git clone git@github.com:kayvank/kinesis-to-aurora.git
cd kinesis-to-aurora
sbt clean compile  ## to build prject

Test & distribution

The following will

• test
• dockerize
• publish docker images
• create executable scripts

sbt test
sbt clean compile universal:packageBin
sbt clean compile docker:stage  docker:publishLocal
docker images | grep 'kinesis-to-aurora' ## verify your docker image was published locally
sbt clean compile docker:stage docker:publish

Running the Project locally

Instruction for running the project locally. We are going to use mysql docker image to simulate AWS Aurora RDS

• Assumptions

  • you have created a AWS kinesis stream

• Setup database

cd ./scripts/sql
docker-compose -f ./docker-compose-mysql.yml up -d
mysql -h 0.0.0.0 -u $JDBC_USER -p $JDBC_DB < ./like_events.sql

• Setup statsd & graphite

docker run -d\
 --name graphite\
 --restart=always\
 -p 80:80\
 -p 2003-2004:2003-2004\
 -p 2023-2024:2023-2024\
 -p 8125:8125/udp\
 -p 8126:8126\
 hopsoft/graphite-statsd

TODO

Put the various docker images into a docker-compose

To run the project

sbt clean run 

Deployment

• local deployment

sbt docker clean compile docker:stage docker:publishLocal

docker run \ 
   -p9000:9000 \
   -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
   -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
   -e KINESIS_STREAM_NAME=${KINESIS_STREAM_NAME} \
   -e JDBC_URL=${JDBC_URL} \
   -e JDBC_PASSWORD=${JDBC_PASSWORD} \
   -e JDBC_DIRVER=${JDBC_DIRVER}  \
   q2io/kinesis-to-aurora

Sample Json event

{
  "entity_type": "USER",
  "entity_id": "USUV71400762",
  "user_id": "26985937",
  "action": "UNLIKE"
}

Table structure

mysql> describe like_events;
+-------------|--------------|------|-----|-------------------|-------+
| Field       | Type         | Null | Key | Default           | Extra |
+-------------|--------------|------|-----|-------------------|-------+
| id          | varchar(40)  | YES  | MUL | NULL              |       |
| user_id     | varchar(40)  | YES  | MUL | NULL              |       |
| entity_id   | varchar(140) | YES  | MUL | NULL              |       |
| entity_type | varchar(20)  | YES  | MUL | NULL              |       |
| ts          | timestamp    | NO   | MUL | CURRENT_TIMESTAMP |       |
| created_at  | timestamp    | NO   |     | CURRENT_TIMESTAMP |       |
+-------------|--------------|------|-----|-------------------|-------+

Authors

• Kayvan Kazeminejad - Intitial work

References

• scalaz
• doobie
• circe
• http4s
• cats
• kcl