Skip to content

Latest commit

 

History

History
99 lines (71 loc) · 2.58 KB

README.md

File metadata and controls

99 lines (71 loc) · 2.58 KB
Raw

Flogi

Flogi (Fluentd + logs + Intelligence) in an application to monitor Containers logs and take metrics with a little bit of intelligence (but not yet).

Flogi is built with the EFK stack (Elasticsearch + Fluentd + Kibana) with a metricbeat integration.

We also provide a jupyter-notebook that connects to elasticsearch and applies LDA (Latent Dirichlet allocation) to some of the logs to detect words that are related to each other by topics.

Link to the project model canvas.

Architecture

Flogi architecture

Dependencies

  • docker engine version 19.03 or higher
  • docker-compose version 1.27 or higher

Running the Project

  1. Make sure your Docker have enough memory. We recommend at least 4GB. (we set this up for you at the script we provided, but only for linux)

  2. Clone the repo:

git clone https://github.com/lionliu/Flogi.git

  1. Setup fluentd as the logging driver at your docker compose file:

    • Put these lines into the container you wish to monitor:
    logging:
    driver: fluentd
    options:
         fluentd-address: localhost:24224
          tag: docker.{{.ID}} # container-id by default
    • Example:
    version: "3"

services:
    front-end:
        image: image
        restart: always
        read_only: true
        logging:
            driver: fluentd
            options:
                fluentd-address: localhost:24224
                tag: docker.{{.ID}} # container-id by default

  2. Execute the script

bash start.sh

  1. Access Kibana at localhost:5601

  2. To access the metrics:

    • Click the hambuguer button
    • Select metrics at the Observability section
    • At first it will show only the metricbeat container. To view all the containers, click at the Show menu and select Docker Containers.

  3. To access logs:

    • Click the hambuguer button
    • Select discover

Accessing the clusterizer

  1. After running the steps above, get the clusterizer address with the following command:
docker logs clusterizer

Stopping the Project

  1. Run: docker-compose down

Built with

  • Fluentd
  • Elasticsearch
  • Kibana
  • Metricbeat
  • Docker
  • jupyter notebook

Link to the project article (in portuguese)

https://docs.google.com/document/d/1_QJ-v5JtKj_vwdGG-jf4hQ2P8u4Gq_Fcojd-hw5BFvw/edit?usp=sharing

Team

  1. João Lira: jpls@cin.ufpe.br
  2. José Reginaldo: jrbj@cin.ufpe.br
  3. Leão Liu: llm2@cin.ufpe.br