Mobile Push with Kafka

The objective of this project is to demonstrate the use of queues to integrate services of a system, avoiding the strong coupling between them. As an example, an integration with Firebase will be used to send push messages to an Android application.

This document describes the steps needed to build and run the Mobile Push Sender using Kafka.

Updated: 02 Fev 2021

Table of Contents

• Architecture
• How to run the project
• Developer Environment
• GoLang Projects
  • Prerequisites
    • Linux Dependencies
    • macOS X Dependencies
  • How to build the GoLang project
  • How to optimize the executable size
  • Build from Docker Container (GoLang Container)
  • Project Command Line
  • Server Framework and Graylog Logrus Hook
• Kotlin Projects
  • How to build the Kotlin project
• Java Projects
  • How to build the Java project
• Kafka
• Prometheus
  • Prometheus Container
  • Prometheus Architecture
• Grafana
• Graylog

Architecture

How to run the project

All docker images can be created by docker-compose. To start all containers, use the docker-compose up command. This command will build all the necessary images and create the containers.

Add the following line in hosts file

127.0.0.1   kafka

Below are the hosts and ports for the all containers (docker on localhost):

• ZooKeeper - http://localhost:2181
• Kakfa - http://localhost:9092
• Kafdrop - http://localhost:19000
• Prometheus - http://localhost:9090
• Grafana - http://localhost:3000
• Graylog - http://localhost:9000

This project needs at least 4 GB of RAM. On MacOS, docker must have 4G RAM to run all the containers in this project. The ElasticSearch container can exit with code 137 if there is no RAM available.

Developer Environment

Tools

• Docker (https://www.docker.com/products/docker-desktop)
• Go (1.15.7 - https://golang.org/dl)
• GoLand (Optional - https://www.jetbrains.com/go/download)
• Intellij IDEA (Optional - https://www.jetbrains.com/pt-br/idea/download)
• Visual Studio Code (Optional - https://code.visualstudio.com/Download)
  • Visual Studio Code Extensions for Go
    • Go for Visual Studio Code (Microsoft - https://github.com/Microsoft/vscode-go.git)

GoLang Projects

project-name = Procuder or Consumer

Prerequisites

Linux Dependencies

• Ubuntu / Debian: sudo apt-get install golang gcc libgl1-mesa-dev xorg-dev
• Fedora: sudo dnf install golang gcc libXcursor-devel libXrandr-devel mesa-libGL-devel libXi-devel libXinerama-devel libXxf86vm-devel
• Solus: sudo eopkg it -c system.devel golang mesalib-devel libxrandr-devel libxcursor-devel libxi-devel libxinerama-devel
• Arch Linux: sudo pacman -S go xorg-server-devel
• Void Linux: sudo xbps-install -S go base-devel xorg-server-devel libXrandr-devel libXcursor-devel libXinerama-devel

macOS X Dependencies

• Install Xcode from the Mac App Store
• Set up the Xcode command line tools by opening a Terminal window and typing the following: xcode-select --install

How to build the GoLang project

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
    -a \
    -o project-name \
    -ldflags \
    "-s -w \
     -extldflags '-static' \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.BuildTime=$(date -u '+%Y-%m-%d_%H:%M:%S%p') \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.GitCommit=$(git rev-parse HEAD) \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.Version='0.1'" main.go && \
     upx --ultra-brute -v /home/app/build/project-name && \
     upx -t /home/app/build/project-name

How to optimize the executable size

• Install UPX (Ultimate Packer for eXecutables - Compress/expand executable files)

  • MacOS X and Homebrew:

    brew install upx

  • Linux:

    apt-get install -y upx

  • Windows:

    choco install upx

• Compress the server executable with UPX:

  upx --ultra-brute -v ./project-name && upx -t ./project-name

  or

  upx -9 -v ./project-name && upx -t ./project-name

Build from Docker Container (GoLang Container)

The docker-composer.yml file has the build section for all GoLang project. This section uses the Dockerfile below to generate the project's executable and optimize it using the upx tool.

FROM golang:1.15.7-alpine as builder

RUN apk update && \
    apk add --no-cache build-base && \
    apk add --no-cache upx git ca-certificates tzdata && \
    update-ca-certificates && \
    addgroup --system app && adduser -S -G app app

WORKDIR /home/app/build
COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
    -a \
    -o project-name \
    -ldflags \
    "-s -w \
     -extldflags '-static' \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.BuildTime=$(date -u '+%Y-%m-%d_%H:%M:%S%p') \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.GitCommit=$(git rev-parse HEAD) \
     -X github.com/kenniston/mobile-push-kafka/golang/cmd.Version='0.1'" main.go && \
     upx --ultra-brute -v /home/app/build/project-name && \
     upx -t /home/app/build/project-name

FROM scratch

COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group

COPY --from=builder /home/app/build/project-name /home/app/project-name

USER app

ENTRYPOINT ["/home/app/project-name", "run"]

Project Command Line

HTTP Server

The default port can change per project

./project-name run
    --server-port    [Default: 6001      - HTTP Server Port]
    --log-level      [Default: info      - (debug, error, trace, info, warning, panic, fatal)]
    --graylog        [Default: false     - Boolean]
    --graylog-ip     [Default: localhots - Graylog Server IP or host name]
    --graylog-port   [Default: 5555      - Integer]

Desktop Application

./project-name desktop

Server Framework and Graylog Logrus Hook

The Producer project uses a server framework to expose REST endpoints (golang/restserver directory). The server uses Iris Web Framework (https://github.com/kataras/iris/) to expose endpoints through project controllers.

The Logrus Graylog Hook (https://github.com/gemnasium/logrus-graylog-hook) is used to send log message through Logrus to Graylog server. The Hook is configured on the server framework:

var useGraylog = v.GetBool("graylog")
var graylogIP = v.GetString("graylog-ip")
var graylogPort = v.GetInt32("graylog-port")
if useGraylog {
  hook := graylog.NewGraylogHook(fmt.Sprintf("%s:%d", graylogIP, graylogPort), nil)
  logrus.AddHook(hook)
  logrus.Infof("Log messages are now sent to Graylog (udp://%s:%d)", graylogIP, graylogPort)
}

Kotlin Projects

How to build the Kotlin project

Not implemented yet

Java Projects

How to build the Java project

Not implemented yet

Kafka

Kafka is the message broker for this project. Services communicate with each other through Kafka topics. The project uses two topics: the first one is used to send push messages through Firebase component; the second one is used to return the results from Firebase to the caller.

Message Flow

The Kafka image for this project is based on wurstmeister/kafka:2.13-2.7.0 which provides additional parameters for Kafka containers. In addition, the final image also contains the library responsible for sending the monitoring data to prometheus (Prometheus JMX Agent).

With default setup the Kafka container starts with two topics: MobileSendPush and MobilePushResult. Each topic has 4 partition and 1 replica.

The Kafdrop (http://localhost:19000) can be used to manager Kafka topics. Kafdrop provides a visual Dashboard that shows Kafka information (Cluster, Brokers and Topics) and provide a visual topic editor. New topic can be created using Kafdrop, avoiding the Kafka command line tool.

Prometheus

Prometheus Container

This project uses the official Prometheus container to build a new image. The new image has the config file used to link Prometheus with Kafka JXM port to receive the Kafka's metrics.

Kafka uses the Prometheus JMX Agent (port 8082 in this project) to expose Prometheus metrics through his host.

Prometheus Architecture

https://hub.docker.com/r/prom/prometheus

Grafana

This project uses the official Grafana container. After run the project using the docker-compose up the Prometheus's Datasource must be configured. Use the following URL (docker on localhost) to configure the Prometheus Datasource:

http://localhost:3000/datasources/new?utm_source=grafana_gettingstarted

Default user: admin
Default password: admin

Select Prometheus in the Time series database section:

Grafana Datasources

Fill in the fields as the image below:

Prometheus Datasource Info

Last but not least, set up the Kafka Dashboard in Grafana. Import the dashboard file from the kakfa folder (grafana-dashboard-kafka-metrics_rev4.json) using a URL below:

http://localhost:3000/dashboard/import

Import the Kafka's Dashboard using the Upload JSON file button

Grafana - Kafka Dashboard

Graylog

This project provides a visual dashboard for logs using a Graylog container. Clients (Golang producer/consumer, Java producer/consumer and Kotlin producer/consumer) can uses GELF messages to send logs to the Graylog.

To send log messages to Graylog a new Input must be configured. To create a new Input follow the steps below:

Graylog Create Input - Step 1

Select GELF UDP and click on Launch new input button - Step 2

Fill in the GELF UDP Input information (title and port are the most important) and click on Save button - Step 2

With the input information (port and Graylog address) the clients can send log messages to the Graylog. Messages can be filtered on the Graylog Search tab.

Graylog Dashboard - Logs