Skip to content

A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching. The whole application is also dockerized.

License

Notifications You must be signed in to change notification settings

rv404674/goRubu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

goRubu

This repo contains implementation of a Url Shortner written in Go.

Build Status Coverage Status Go Report Card Stars MIT License

Contents

What is goRubu? πŸš€

  1. A Url Shortner written in Go, with MongoDb based backend.
  2. Supports Caching for Hot urls, with Memcached, using a LRU based eviction strategy, and write through type of caching mechanism. Saw 200% decrease in Read Latency for URL redirection, after caching.
  3. Used Travis CI for adding a CI/CD pipeline.
  4. Dockerized the whole application. Used Docker compose for tying up different containers and multi-stage build for reducing the size of docker image by 900%.
  5. Prometheus and Grafana based monitoring, to get an overall picture of the system and application metrics.
  6. Handled Concurrent Requests, using a global Mutex, over Request Level. Use Optimistic Concurrency Control and Pessimistic Concurrency Control, to see if Performance can be increased.
  7. Contains Api Validation and Logging Middlewares, along with Swagger based documentation

Why goRubu? 🐢

Wanted to Learn Go and system design, by building a project. Hence goRubu.

BenchMarking

NOTE - Url Shortner is a read heavy system (read:write = 100:1), and these load tests are done on a single Machine (8Gb Ram, i5 Processor).

Check this out for more info.

  1. For Url Redirection, with 1000 Concurrent Users, bombarding the app server for 2mins:
http_req_duration..........: avg=2.39s    min=0s     med=2.27s max=7.51s    p(90)=3.65s p(95)=4.16s
http_reqs..................: 49587   413.224692/s
  1. For Url Shortening, for same specs
http_req_duration..........: avg=8.6s     min=0s      med=8.6s  max=19.03s   p(90)=10.33s p(95)=10.69s
http_reqs..................: 13345   111.207074/s

Running Server βš™οΈ

Docker

  1. You need to have docker and docker-compose installed. After that just do
git clone https://github.com/rv404674/goRubu.git
cd goRubu
make docker

Note - I haven't dockerized prometheus and grafana with it. Containerized goRubu and locally installed prometheus and grafana will work fine as prometheus is only listening to "local:8080/metrics".

Check the Api's Section afterwards.

Local

Prerequisites βœ…

Ensure you have the following installed. Mongodb Memcached Make

On Macos, simply use thse

$ brew update
$ brew install mongodb/brew/mongodb-community
$ brew install memcached
$ brew install make

Note: After that check whether these have been started, else you will get connection error.

$ brew services list

if any service is not up do

$ brew service start service_name

Then we need to download the tar files for prometheus, grafana and node exporter. We will need to edit their config files, hence we are not using brew install for these. Also it becomes easy to run the server and do changes.

For Grafana

wget https://dl.grafana.com/oss/release/grafana-6.7.3.darwin-amd64.tar.gz
tar -zxvf grafana-6.7.3.darwin-amd64.tar.gz
mv grafana-6.7.3 /usr/local/bin
cd /usr/local/bin/grafana-6.7.3/bin
./grafana-server # this will run the grafana server

Note - By default prometheus server runs on 9090, and grafana on 3000.

Similarly download tar files for prometheus and node exporter and run there servers as well.

Step to Run locally

  1. Go to the dir where prometheus is installed and change the prometheus default .yml file to this one new_yml, run the prometheus server, and node_exporter server.

  2. do

$git clone https://github.com/rv404674/goRubu.git
  1. cd in goRubu, and do
make deps

It will install all the go dependencies.

  1. Then do
make install
make execute

Note: To see what these commands do check out this makefile

Api's πŸ’»

  1. Hit localhost:8080/all/shorten_url with any url as key.
{
	"Url": "https://www.redditgifts.com/exchanges/manage"
}

The endpoint will return a shortened URL.

  1. Hit http://localhost:8080/all/redirect with the shortened url to get the original URL back.
{
	"Url": "https://goRubu/MTAxMTA="
}

Monitoring πŸ”¬

Working:

  1. Prometheus follows a Pull based Mechanism instead of Push Based.
  2. goRubu exposes an HTTP endpoint exposing monitoring metrics.
  3. Prometheus then periodically download the metrics. For UI, prometheus is used as a data source for Grafana.

Grafana on Top of Prometheus

Contributing 🍻

Peformance Improvements, bug fixes, better design approaches are welcome. Please Discuss your change by raising an issue, beforehand.

Future Todo πŸ““

  1. Use Promtheus and Grafana as Docker Containers instead of Local Installation.
  2. Use Kubernetes instead of Docker Compose.
  3. Increase Coverage to more than 90%.
  4. Add Monitoring in Prometheus for Number of 2xx, 3xx and 4xx.

Maintainer 😎

Rahul Verma Linkedin Email

License πŸ“œ

MIT Β© Rahul Verma