The free online encyclopedia Wikipedia says: "The term Network Monitoring describes the use of a system that constantly monitors a computer network for slow or failing systems and that notifies the network administrator in case of outages via email, pager or other alarms."
For any company to grow and be successful, it is essential the it establish a method of monitoring its network activity is essential to. This should include solutions that can automatically detect and respond to threats and performance issues in real time, as well as predict possible issues in the future.
The object of this project is to add the monitoring infrastructure to the data engineering pipeline. This provides the reporting capability on the capacity and performance of the hardware and applications so you can know when you are running into issues as well use it for forecasting future needs.
Identifying what needs to be monitored
- Creating a “visual” map of the most critical parts of the network
- avoiding information overload
Deciding how to measure what is being monitored
- Setting base lines
Ingestion: AWS S3
Data Processing: Spark
User Interface (UI): Flask
Monitoring System Prometheus
- Containerize the monitoring system
- Add caching to the system
[How to use prometheus ] (https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-ubuntu-14-04-server)
[ How to Install and Secure Grafana] (https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-16-04)
[Monitoring MySQL] (https://dzone.com/articles/monitoring-mysql)
Monitoring Read/Write Latency