Lightstream

Photo hosting microservice application
Report Bug · Request Feature

Table of Contents

1. About The Project
   • Project Structure
   • Monitoring
   • Continuous Integration
2. Getting Started
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. Contact

About The Project

This project is a photo hosting implementation aimed at learning microservice architecture and devops culture.

Technologies used:

• Golang
• PostgreSQL
• gRPC, REST, AMQP (RabbitMQ)
• Docker, CI/CD, Github Actions, Terraform, Kubernetes
• Amazon S3, Yandex Cloud
• NGINX
• Prometheus, Grafana
• Python, JS, HTML, CSS

Project Structure

The project consists of three microservices: gateway, history, storage:

• gateway microservice is the central part of the application. All other parts of the application are associated with this service.
• history microservice saves the history of image requests.
• storage microservice communicates with Yandex cloud and saves pictures there.

(back to top)

Monitoring

The project is configured to monitor the system with Prometheus and Grafana.
(Grafana - http://localhost:3030)

4 Golden Signals are used to monitor the application

If the server crashes or the load is too high, the Alertmanager will send a notification in Telegram.

(back to top)

Continuous Integration

Continuous Integration pipeline set up in the project.
In the case of a push to the main branch, Docker Images of changed microservices are rebuilt and sent to the Yandex Cloud Registry

(back to top)

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these example steps.

Installation

1. Clone the repo

   git clone https://github.com/rvinnie/lightstream

2. Set environment variables
   • services/gateway/.env

     POSTGRES_USER=<postgres user>
     POSTGRES_PASSWORD=<postgres password>
     POSTGRES_DB=<database name>
     DATABASE_HOST=postgres_gateway
     
     RABBIT_USER=<RabbitMQ user>
     RABBIT_PASSWORD=<RabbitMQ password>
     
     GIN_MODE=debug

   • services/history/.env

     POSTGRES_USER=<postgres user>
     POSTGRES_PASSWORD=<postgres password>
     POSTGRES_DB=<database name>
     DATABASE_HOST=postgres_history
     
     RABBIT_USER=<RabbitMQ user>
     RABBIT_PASSWORD=<RabbitMQ password>

   • services/history/.env

     AWS_ACCESS_KEY_ID=<Yandex Cloud access key identifier>
     AWS_SECRET_ACCESS_KEY=<Yandex Cloud secret access key>
     AWS_REGION=ru-central1

   • monitoring/alertmanager/alertmanager.yml

     ...
     - bot_token: <telegram bot token (string)>
       api_url: 'https://api.telegram.org'
       chat_id: <telegram chat id (int)>
     ...

   • .github/workflows/push.yml
     Put YC_SA_JSON_CREDENTIALS in Github Actions secrets.
3. Make sure docker is installed
4. Choose one of the three versions of the project and run
   • development version

     make

   • development version with stress testing

     make version=stress

   • production version

     make version=prod

(back to top)

Usage

After launch, go to the address in the browser https://localhost

Here you can add an image to storage, find an image by id and get the whole gallery of images. All added images are saved in Yandex Cloud Object Storage.

An example of a gallery with two images (Upload two images -> press Get all)

(back to top)

Roadmap

• Send image by link
• Add user entity
• Add authorization microservice (JWT)
• Add Terraform
• Add Kubernetes

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact

Alexey Fedoseev - @fedoseev_alexey - rv1nnie@yandex.ru

Project Link: Lightstream

(back to top)