Implement deployment traceability #361

placek · 2024-03-04T07:48:35Z

As a QA tester,

I want to have a method to trace the information about the deployments,

So that I can have a clear trace of which versions of the modules are deployed on which environments, including a log of the changes.

Acceptance Criteria

Deployment Information Propagation: After each deployment, the system automatically updates a dashboard with information about the deployment, including the environment, version deployed, and deployment timestamp.
Change Log Traceability: A comprehensive log of changes for each version is maintained and easily accessible. This log should include version IDs, environment names, and the date/time of the deploy.
Version Query Capability: Developers and operations teams can easily query which version of a module is deployed in any given environment through a simple interface or command.

placek · 2024-03-04T07:54:03Z

This is a part of the former splitted task #320.

I planned to use pushgateway to propagate the deployment events to prometheus, but it seems there's an issue with setting up the pushgateway service properly under current stack.

So the plan is to use Loki in order to trace deployment changes as events logged on the target machine.

Loki would be also helpful to be used by QA/DevOps in other parts of the system, tracing other events, like cardano node progress or db sync synchronisation status.

This commit introduces Loki and Promtail to the deployment stack, enhancing the project's logging infrastructure by enabling advanced log aggregation, visualization, and analysis capabilities. The integration involves modifications across multiple configuration files and the introduction of new templates for Loki and Promtail. Here's a breakdown of the changes: - config.mk Adjustments: Updated the prepare-config target to include steps for preparing Loki and Promtail configuration files, ensuring these services are configured as part of the deployment setup. - Docker Compose Template Update: Extended the docker-compose.yml.tpl to define loki and promtail services, including their respective Docker images, volumes, and dependencies. This update ensures that both Loki for log aggregation and Promtail for log shipping are part of the containerized environment. - Grafana Data Sources Configuration: Modified the grafana-provisioning/datasources/datasource.yml to include Loki as a data source, enabling Grafana to query log data collected by Loki. This allows for comprehensive log visualization alongside metrics within Grafana dashboards. - Loki Configuration Template (loki.yml): Introduced a new configuration template for Loki, specifying its operational parameters such as server ports, ingester configuration, schema, storage, and retention policies. This configuration is designed to optimize Loki's log aggregation capabilities within the deployment environment. - Promtail Configuration Template (promtail.yml): Added a new configuration template for Promtail, detailing how log files are located, scraped, and forwarded to Loki. The configuration includes the definition of job names, static configs, and labels for log categorization. The integration of Loki and Promtail represents a significant enhancement to the project's observability infrastructure. It enables efficient log collection, storage, and querying, providing developers and operators with powerful tools for troubleshooting, performance monitoring, and operational analysis.

This commit enhances the infrastructure setup process by incorporating an initialization step for a deployment log within the EC2 instance's user data script. This change is aimed at bolstering deployment traceability and logging mechanisms for the GovTool project. Key aspects of the update include: - Deployment Log File Creation: Added commands to create a new log file, /var/log/deployment.log, during the initial setup of the EC2 instance. This log file is intended to capture and store deployment-related events, making it easier to track and diagnose issues that may arise during or after the deployment process. - Log File Ownership Adjustment: The ownership of the newly created deployment log file is set to ubuntu:ubuntu, ensuring that the application, running under the ubuntu user, has the necessary permissions to write deployment events to the file. This modification directly supports the project's objective to enhance deployment traceability and operational monitoring by providing a dedicated mechanism for logging deployment activities, thereby facilitating better insight into deployment processes and outcomes.

…process - Modified 'all' target in Makefile to exclude 'info' and directly include 'notify', simplifying the deployment sequence. - Introduced 'log-deployment' in info.mk for logging deployment details to /var/log/deployment.log. This includes capturing essential deployment metadata such as the environment, branch, commit, and versions of the deployed components, formatted with a timestamp and marked as INFO for clarity. - Updated 'notify' target to depend on 'info' and 'log-deployment', ensuring comprehensive deployment information is shared and recorded. This includes deployment success notifications with detailed component versions and source information. - Adjusted the format for logging deployment information to improve readability and consistency in monitoring tools. These changes enhance the project's deployment traceability, allowing for better tracking of deployments and easier debugging and auditing of deployed versions.

- Added a new panel to the GovTool Grafana dashboard configuration for visualizing deployment logs. - This panel uses the Loki data source to display logs from `/var/log/deployment.log`, providing a trace of every deployment made. - The panel is configured to display logs in descending order, focusing on deployment events. This aims to facilitate quick identification of deployment activities and troubleshooting issues. - Updated the dashboard version to reflect these changes, ensuring that the dashboard remains up-to-date and functional with the latest configurations. This enhancement improves the observability of the deployment process, making it easier for operators to monitor deployment activities and investigate issues directly from the Grafana interface.

placek · 2024-03-06T09:36:21Z

https://dev-sanchonet.govtool.byron.network/grafana/d/e3697f62-6a01-4596-b626-d46141441b32/govtool-basic-stats?orgId=1&refresh=10s