Skip to content

Latest commit

 

History

History
381 lines (273 loc) · 12.9 KB

READMEv2.md

File metadata and controls

381 lines (273 loc) · 12.9 KB
Raw

Build Contributors Forks Stargazers Issues MIT License


Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Documentation
  5. Roadmap
  6. Contributing
  7. License
  8. Contact
  9. Acknowledgments

About The Project

Implement rate limiting to study the the go programming language and learn things like random replacement, consistent hashing, leaky bucket, ... .

(back to top)

Built With

The project is written entirely in Golang.

The gorilla mux framework is used for http routing and implementing the handlers that enforce rate limiting before a request is forwarded to the service.

The package validator is used to validate the service's json request using struct's and field tags.

For monitoring, the prometheus client package is used to expose the service metrics.

The package prometheus-middleware defines the two important prometheus metrics for http based services. These are the latency and the number of requests per status code, method and path.

Test With

The following packages are used only in the unit tests.

(back to top)

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

The following packages/tools has to be available.

Installation

Below is an example of how you can instruct your audience on installing and setting up your app. This template doesn't rely on any external dependencies or services.

  1. Clone this repo 
    git clone git@github.com:fr123k/fred-the-guardian.git
#or
git clone https://github.com/fr123k/fred-the-guardian.
cd fred-the-guardian.

Command Line Interface (pong)

Build

# build the pong command line interface
make build-cli

Run

# show the pong cli argument options
./build/pong -help
Usage of ./build/pong:
  -auto
        use auto discovery of possible ping services (Default: false)
  -path string
        root path of the ping service (Default: /) (default "/")
  -port string
        port of the ping service (Default: 8080) (default "8080")
  -rndsec
        set true to generate a random secret for each request (Default: false)
  -secret string
        specify the secret value for the X-SECRET-KEY http header (Default: top secret) (default "top secret")
  -server string
        server address of the ping service (Default: 127.0.0.1) (default "127.0.0.1")

# run with default ping service endpoint localhost:8080/
./build/pong

# run with auto discovery try couple of options to reach a ping service
./build/pong -auto true

# run with a specific ipaddress and port defintion for the ping service endpoint
./build/pong -server 192.168.2.45 -port 8888 -path /proxy/

Service (ping)

Build

# build the go binary
make build
# or build the go binary within a docker image
make docker-build

Run

# run the go binary
make run
# or start the ping service docker container in detach mode
make docker--run

Test

make test
# or if the default port isn't available
PORT=8888 make test
# or just run the curl command yourself adjust the port as needed
curl -X POST -H 'X-SECRET-KEY:top secret' -v http://localhost:8080/ping

Deployment

The deployment has only been tested with a local minikube vm so far. The deployment files are located under 'deploy/local'.

There is a [Makefile] in the local deployment folder.

cd deploy/local
# The vagrant make target will download the fr123k/ubuntu21-minikube vagrant box from vagrant cloud and 
# provision it based on the VagrantFile.
# The vagrant and the setup target steps can be skipped if you have aleady a local kubernetes cluster.
export KUBECONFIG_SAVE=${KUBECONFIG}
export KUBECONFIG=$(pwd)/kubectl.config
EXTERNAL_IP=172.28.128.16 make vagrant setup
# this just run kubectl apply for the ping k8s manifest files
make deploy
# this will use curl to send the ping request to the k8s pod
# if you use a different k8s cluster then the vagrant minikube one then
# adjust the EXTERNAL_IP
EXTERNAL_IP=172.28.128.16 make test
export KUBECONFIG=${KUBECONFIG_SAVE}

(back to top)

Usage

Use this space to show useful examples of how a project can be used. Additional screenshots, code examples and demos work well in this space. You may also link to more resources.

For more examples, please refer to the Documentation

(back to top)

Documentation

Counter

Bucket

Random Replacement

(back to top)

Roadmap

Version 1

  • Implementation of the first simple rate limiting based on a counter in memory.
    • single thread safe counter
    • simple rate limiting with counter reset
    • data structure for multiple counters

Version 2

  • Implementing the ping web server API without rate limiting
  • Implementing the pong client interface
  • Setting up a local minikube for the first deployment

Version 3

  • Implement rate limiting in the ping service with counters stored in memory.
  • add ping service test
    • Implementation of edge case and happy path tests
    • Implementation of rate limiting tests
  • refactoring the ping service, reducing the lines of code and abstracting the general behavior.
  • add pong cli test
  • implement pong cli handling of rate limiting responses.
  • refactoring pong cli, reducing lines of code and abstracting general behavior.
  • Add bucket count and memory usage to status endpoint response
  • add prometheus metrics instrumentation for monitoring
  • Implement eviction of bucket counters to purge memory of unused counters
  • Update documentation to the latest implemented features.

Version 4

  • Implement rate limiting with counters stored in redis.

Version 5

Version 6

Version 7

Version 8

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

Project Link: https://github.com/fr123k/fred-the-guardian

(back to top)

Acknowledgments

The following articles and resources provide useful insights into rate limiting and learning the Go programming language.

(back to top)