Godays 2019 - Painless stream processing with Go and Kafka

Slides available here.

Agenda

• Checkout the data
• Setup working environment
• clone the repo
• tasks
  1. Consume events using Sarama
  2. Track and view state of taxis
  3. alert on license fraud
  4. detect fraud
  5. extras (testing, monitoring, visualization of coordinate-cells)

Set up your working environment

• Golang https://golang.org/dl/
• Docker/Git
  • Mac:

brew install git
brew cask install docker
open /Applications/Docker.app

• Linux:

sudo apt-get install docker git

• Setup a Gopath (if you don't have it yet)

mkdir -p $HOME/gocode/
cd $HOME/gocode
export GOPATH=$HOME/gocode

• Get the Workshop Code

# need to have $GOPATH set
mkdir -p $GOPATH/src/github.com/frairon/goka-godays2019
git clone https://github.com/frairon/goka-godays2019.git
# OR
go get github.com/frairon/goka-godays2019

• Get the workshop data

# tiny data set is located in testdata/taxidata_tiny.csv
# 100k dataset can be loaded from https://storage.googleapis.com/lv-goka-godays2019/taxidata_100k.csv
make get-100

# [Optional]
# full dataset (~320MB) can be downloaded from https://storage.googleapis.com/lv-goka-godays2019/taxidata_complete.csv
make get-complete

• Start Kafka locally

make restart-kafka

Now you're ready to start working with the code