linkerProcessorSample Intro

A sample code for processing infomation from Kafka and store to Cassandra using Apache Spark

Tag introductions

###tag v0.1 realizations: 2. collect info from kafka topic 3. calculate cpu total usage as well as each cpu usage 4. save cpu usage to cassandra 5. test successfully on Mesos cluster with Spark client mode

###tag v0.2

1. the same as v0.1
2. calculate cpu_all usage

###tag v0.3

1. the same as v0.1
2. realize calculating muli-machine cpu usage
3. build spark driver image and executor image for Mesos Cluster

Usage

Prerequisite

First, you should set up zookeeper, cassandra and kafka broker. Then create kafka topic and cassandra keyspace

Second:

1. download Apache Spark
2. run linkerConnector
   • git clone https://github.com/LinkerNetworks/linkerConnector.git
   • install Golang
   • cd path/to/linkerConnector
   • go build and go install
   • linkerConnector -i 1000 -d kafka -t -s
3. git clone linkerProcessSample(https://github.com/adolphlwq/linkerProcessorSample.git)

submit code to Spark

Please refer Here for detail

Reference

• Apache Spark Advanced Dependency Management
• Maven central repo
• calculate cpu usage in Golang
• Linux Kernel about proc
• Cassandra tutorial
• Cassandra data types
• Cassandra user user-defined-type

TODOs

• collect info from kafka
• save processes info to cassandra
• save machine info to cassandra
• save kafka message to cassandra directly
• overall cpu usage from linkerConnector (via Kafka)
  • calculate cpu usage and save to cassandra
  • calculate cpu usage from multi machine
• Mesos agent usage from linkerConnector (via Kafka)
• build Mesos Spark executor docker image for testing code on Mesos cluster
  • Mesos Spark executor beta2 Dockerfile
• research Spark streaming's "window" and improve code