zenly-task

Task description

Getting Started

Requirements:

• Docker
• Docker Compose

Optional:

• GoLand for running prepared configurations (load testing tool, application with grpc service)
• protoc, protoc-gen-go-grpc for generating protobuf implementations
• plantuml for generating architecture diagram

Start

Start all (infrastructure, monitoring, application):

make up

OR

Start infrastructure (ZooKeeper, Kafka, NATS):

make infra-up

Start monitoring (Prometheus, Prometheus Exporters, Grafana):

make monitoring-up

Start application:

make app-up

Stop

Stop all

make down

OR

Stop infrastructure:

make infra-down

Stop monitoring:

make monitoring-down

Stop application:

make app-down

Architecture Overview

Key Points

Architecture

• NATS uses 1 subject per 1 user
• gRPC publish stream calls enrichers and publishes the enriched geolocation to the NATS subject
• Each enricher has 100ms to complete and returns SetFunc to set the value to the geolocation if the context isn't timeout-ed yet
• Enrichers are supposed to use services to get external data (e.g. weather enricher uses weather service to get fake weather data at location)
• gRPC subscribe stream subscribes to multiple NATS subjects using MultiSub with message handler sending all incoming enriched geolocations to client

Scalability

All parts of the system can be scaled easily:

• Vertical scaling through spawning more application instances isn't needed, since golang eats all the resources available
• Horizontal scaling can be done through setting up more pods (in terms of k8s) and a simple load balancer. Using a load balancer like Linkerd that allows balancing requests across connections isn't that useful because we use gRPC streams to send and receive geolocations, thus it's not a good idea to reopen a stream every time
• NATS can be put into a cluster to have more throughput and less downtime, then each server in a cluster should be added to the app through -nats-servers flag
• More kafka brokers can be added through -kafka-brokers flag

Monitoring:

• Enriching process reports each enricher's time with result (in time / timeout) and total enriching time with finish reason (complete / timeout)
• Grafana lives at http://localhost:3000/ with username admin and password admin

Time to Check It Works!

1. Start infrastructure, monitoring and application

   make up
   

2. Generate infinite load with 10 RPS for 1 publish stream for 10 users and listen on 10 subscribe streams

   make load
   

3. Head to http://localhost:3000 with admin:admin, find Zenly dashboard in search and see graphs are filling with data
4. Stop loader with Ctrl-C and see stats printed out