Repository navigation
Releases: max-romagnoli/System-Telemetry-Agent
Releases · max-romagnoli/System-Telemetry-Agent
Release list
v4.0.0
Release Notes
This is the fourth version of ST-Agent. It features:
- The finished dashboard visualisations, alerting system and a small server to simulate different machine workloads through stress scripts.
- All the services of our system set up locally, fully containerized and orchestrated with Docker compose. The local setup supports monitoring for 1 to 30 virtual machines.
- An Azure cloud setup with an Azure Scale Set of N Virtual Machines all running the ST Agent, and a deployed instance of Prometheus.
Run Instructions
- To run the app locally with N targets follow the instructions in
docker/README_docker.md - To monitor the VM on Azure, start Docker compose and the instances to monitor on Azure portal or Azure CLI.
Exporter
- Refactored exporter.py to set CPU by core gauges in a separate method for better scalability.
- Add class variable in Network collector to fix Network metrics not being returned correctly.
Prometheus
- Created an Alertmanager directory.
- Connected Prometheus to Alertmanager.
- Added configuration files to dispatch emails based on rules.
- Added rules that trigger when a metric passes a specific tresholds.
- Added rules that trigger when an instance is down.
- Added predictive rules that trigger when a metric is projected to reach a certain level in a specified amount of hours.
Dashboard
- Rolled out the final version of the homepage System Overview.
- Added the following visualisations to the homepage:
- Disk Utilization Overview Panel: aggregates Disk utilization metrics from all machines monitored.
- RAM Utilization Overview Panel: aggregates RAM utilization metrics from all machines monitored.
- Network In/Out Utilization Overview Panels: aggregates Network utilization metrics from all machines monitored.
- Changed all panels to show threshold limits.
- Created visualisations to display Prometheus alerts
- Created a visualisation to display notes taken on the dashboard.
- Added a clock visualisation
- Added logos to page headers
- Removed CPU temperature panels in Machine Drilldown dashboard
- Added two timeseries charts for logical and physical cores in the Machine Drilldown dashboard
Simulation Tool
- Rolled out a simple simulation tool to interact with the scripts that simulate different workloads.
- Added a simple HTTP server to handle running the scripts.
- Added a simple frontend with buttons to start the simulations.
Azure
- Connected the Azure Scale Set to both the local and cloud Prometheus instance.
- Added a script to simulate heavy and normal workloads on Azure machines
- Made the deployed exporters all start on VMSS Machine startup
- Added workflow file for CD to Azure VMs
Docker
- Added an alertmanager service to Docker compose.
- Added a script to simulate one of the Docker containers shutting down.
- Refactored Prometheus local config files to discover however many targets have been created. This was achieved using File-based Service Discovery.
- Refactored Prometheus local config files to scrape metrics from deployed targets on Azure.
Misc
- Create a script for Alertmanager to append Gmail App Passwords from environment variables to the config files.
- Updated documentation ahead of final release.
v3.0.1
Release Notes
This is the third version of ST-Agent. It features:
- All the services of our system set up locally, fully containerized and orchestrated with Docker compose. The local setup supports monitoring for 1 to 30 virtual machines.
- An Azure cloud setup with 1 Virtual Machine deployed on Azure and monitored locally, an Azure Scale Set of N Virtual Machines, and a deployed instance of Prometheus.
Run Instructions
- To run the app locally with N targets follow the instructions in
docker/README_docker.md - To monitor the VM on Azure, also follow the instructions in
conf/SSH-connection.md
Exporter
- Added a method to collect CPU utilization metrics by logical and physical core
- Added two methods to collect outbound and inbound traffic rates in megabits per second
- Checked for none values in RAM and Network collector methods.
- Added more unit tests for network metrics.
- Refactored string representation tests to check for format using regex.
- Fixed a bug that caused the string representation of CPU collector not always pass in the CI pipeline.
Dashboard
- Rolled out an early homepage dashboard for our agent.
- Added the following visualisations to the homepage:
- RAM Utilization Overview Panel: aggregates RAM utilization metrics from all machines monitored.
- Added data links to allow user to click on one machine in the homepage and be sent to the machine dashboard for that machine.
- Added dashboard links to allow user to go back to the homepage.
- Adjusted style and colour palette by reducing the number of colour on the dashboards.
- Adjusted data thresholds according to target machine typical performance and limits.
- Fixed a bug in a PromQL query that caused the RAM donut chart to display bytes used instead of RAM utilization percentage.
Azure
- Deployed an Azure Scale Set of 5 virtual machines and exposed ports to export metrics.
- Deployed an instance of Prometheus on a VM
- Manually deployed Prometheus binaries and configuration to scrape from one target in the Azure Scale Set.
- Created a systemd service on the Prometheus VM to run the Prometheus server.
Local Development
- Added a shell script to override docker-compose and generate 1 to 30 Ubuntu Docker containers for local testing.
- Refactored Prometheus local config files to discover however many targets have been created. This was achieved using File-based Service Discovery.
- Refactored Prometheus local config files to scrape metrics from deployed targets on Azure.
Misc
- Added documentation for local N-machines setup:
v3.0.0
Release Notes
This is the third version of ST-Agent. It features all the services of our system set up locally, fully containerized and orchestrated with Docker compose as well as an initial cloud setup with 1 Virtual Machine running on Azure and monitored locally.
Run Instructions
- To run the app locally follow the instructions in
docker/README_docker.md - To monitor the VM on Azure, also follow the instructions in
conf/SSH-connection.md
Exporter
- No changes here
Dashboard
- Changed the machine dashboard to display N machines.
- The user can now click on a drop-down menu and monitor the target machine of choice.
- Added template variables to allow the above functionality.
Azure
- Created one Virtual Machine on Azure.
- Manually deployed the Python exporter in the VM.
- Exposed ports to allow Prometheus to read data from the VM.
Configurations
- Configure Prometheus to scrape metrics from the Virtual Machine on Azure alongside the local containers.
Misc
- Updated documentation for Azure setup:
- Added instructions on how to setup a VM on Azure.
- Added instructions on how to manually deploy exporter binaries on Azure.
- Added instructions on how to connect to VM via SSH.
v2.0.1
Release Notes
This version added a few minor changes to v2.0.0 (read here the previous release notes).
Changes
- Added collector methods and tests for Disk metrics (@dalesv)
- Added CPU frequency and temperature visualisations in the dashboard
- Switched the whole dashboard to stagent metrics
Contributors
dalesv
Assets 2
v2.0.0
Release Notes
This is the second version of ST-Agent. It features all the services of our system set up locally, fully containerized and orchestrated with Docker compose.
Run Instructions
- To run the app locally follow the instructions in
docker/README_docker.md
Exporter
- Metrics are collected from an Ubuntu container using our own Python exporter.
- Added collectors methods for the following metrics:
- CPU: utilization, frequency, temperature (@mccooeyc11)
- RAM: utilization, total memory installed (@l3331l4)
- Network: traffic in, traffic out (@cindyariyo)
- Added unit tests for each of the above, fully integrated in the CI pipeline.
Configurations
- Configure Prometheus to scrape metrics from our exporter, alongside Node Exporter.
- Migrated to a fully-containerized development environment to ensure consistent testing.
- Reconfigure both Grafana and Prometheus to work with Docker.
Dashboard
- Rolled out the machine dashboard which displays metrics pulled from Prometheus about one VM.
- Added the following visualisations to the dashboard:_
- CPU panel (@BBBBliW)
- Overall utilization (gauge)
- Utilization by core (timeseries chart)
- Percentages of busy CPU (bar chart)
- Disk panel (@BBBBliW)
- Used vs. Free (pie chart)
- Capacity overtime (timeseries chart)
- Memory (@ife0ye)
- Used vs. Unused (pie chart)
- Usage overtime (timeseries chart)
- Network (@ife0ye)
- Network speed (gauge)
- Traffic In (timeseries chart)
- Traffic Out (timeseries chart)
- CPU panel (@BBBBliW)
- Added first Information Components to the dashboards (headers, hoverable descriptions...).
Misc
- Added a Python script that uses the Grafana API to sync changes made in the Grafana GUI with the local config files versioned in Git.
- Updated documentation to facilitate migration to Docker setup.
Contributors
ife0ye, mccooeyc11, and 3 other contributors
Assets 2
v1.0.0
General
- Initial local development of backend and frontend systems: built out project structure and infrastructure for the system to run locally.
- Integration of Prometheus and Grafana: in this version, our system is targeted to scrape metrics from an Ubuntu container running Node Exporter. Metrics are then pulled from Grafana using PromQL and fed into our dashboards.
- An initial setup of our custom Python exporter is also present and running locally on a Docker container.
- Implemented CI/CD workflows (pytest).
- Instructions on how to run the app locally are in:
docker/README_Docker.md: monitoring agentgrafana/README.md: dashboardprometheus/README.md: Prometheus server