Github actions enhancements - streamlining workflows, workflow deploy automation, configuration refactoring, and bug Fixes #547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… configuration files As a developer managing deployment configurations, the aim is to introduce a generic target that facilitates the creation of a directory structure for configuration files. This adjustment is essential for enabling enhanced control over the configuration file creation process through a more file-centric approach. - Refactored the `config.mk` file within the `govtool` scripts directory. - Replaced individual directory creation commands with a generic target to create necessary subdirectories for configuration files using actual file targets. - Introduced `config_subdirs` and `target_config_subdirs` variables to manage the directory structure more effectively. - Modified existing targets such as `generate-docker-compose-file`, `enable-prometheus`, `prepare-dbsync-secrets`, `prepare-backend-config`, `prepare-prometheus-config`, `prepare-promtail-config`, `prepare-loki-config`, `prepare-grafana-provisioning`, and `prepare-nginx-config` to utilize these new file targets efficiently. The adjustments ensure streamlined directory creation for configuration files, enhancing the overall configurability and ease of maintenance in the deployment process. The introduction of a generic target promotes consistency and clarity in managing configuration files within the deployment environment.
As part of the task to replace phony targets with real file targets in the deployment makefiles, this commit specifically focuses on transforming the `prepare-nginx-config` phony target into actual file targets. The `prepare-nginx-config` rule now explicitly defines the creation of the `auth.conf` file and `govtool.htpasswd` file within the `nginx` directory, improving the declarative nature of configuration generation. Changes: - Updated `config.mk` within the `scripts/govtool` directory - Replaced the `prepare-nginx-config` phony target with the creation of `auth.conf` and `govtool.htpasswd` files as actual file targets in the `$(target_config_dir)/nginx` directory - Conditional logic based on the `domain` variable now controls the content generation for `auth.conf` and `govtool.htpasswd` files, ensuring proper file definition during the nginx configuration setup.
As a part of refactoring deployment makefiles to use file targets instead of phony targets, this commit focuses on updating the prepare-loki-config target. The switch from a phony target to a file target enhances the control over configuration generation. Changes: - The prepare-config makefile target has been modified to include the $(target_config_dir)/loki.yml file target in its dependency list. - The prepare-loki-config phony target has been replaced with a file target declaration, ensuring that the loki.yml file is correctly copied to the target directory. - The prepare-grafana-provisioning target now excludes the prepare-loki-config target as a dependency due to the transition to file-driven targets. The refactorization of the prepare-loki-config target aligns with the overarching goal of enhancing declarative control over the deployment configuration generation process. This change streamlines the makefile structure, improving the clarity and specificity of the targets involved in the deployment configuration setup.
As a developer handling deployment makefiles, the refactor aims to enhance configuration management by substituting phony targets with actual file targets, thereby enabling a more definitive control over the configuration generation process. The commit involves modifying the `config.mk` file in the `scripts/govtool` directory. Specifically, the `prepare-promtail-config` target is altered by changing it to a file target instead of a phony target. This change ensures that the `promtail.yml` file is directly produced within the target configuration directory, aligning with the goal of transitioning to file-driven processes in deployment management for improved declarative control.
As a developer taking care of deployment configurations, the `prepare-prometheus-config` target needed to be upgraded as part of the transition from phony targets to file targets. This refactor enables a more file-driven approach to configuration generation by implementing `prepare-prometheus-config` as an actual file target within the `config.mk` file. Changes: - Updated the `prepare-prometheus-config` target in `config.mk` to be a file target, ensuring its functionality remains intact. - Replaced the use of phony targets with actual file targets for `prometheus.yml`, `promtail.yml`, and `loki.yml` within the deployment makefiles to enhance declarative control over configuration generation.
Refactoring the `prepare-backend-config` target within the `config.mk` makefile to improve deployment configuration generation by using actual file targets instead of phony targets. This change enhances the declarative nature of the deployment process, ensuring better control over configuration outcomes. Changes: - Modified `prepare-config` target in `config.mk` to include `$(target_config_dir)/backend-config.json` as an actual file target instead of a phony target. - Updated the `prepare-backend-config` rule to generate the `backend-config.json` file directly from the template and populate it with necessary values based on environment variables. - Refactored the makefile to shift towards a more file-driven approach for configuration generation, ensuring that specific actions remain as phony targets for execution purposes.
…ol.htpasswd` In the `scripts/govtool/config.mk` file, the `prepare-config` target was updated to include `$(target_config_dir)/nginx/govtool.htpasswd` file within the list of outputs. Additionally, the dependency of `$(target_config_dir)/nginx/auth.conf` target was modified to include only `$(target_config_dir)/nginx` directory, removing the `govtool.htpasswd` file as a prerequisite for this target. This change aligns the target declarations more accurately with the actual file dependencies and ensures that `govtool.htpasswd` is now appropriately listed as an output of the `prepare-config` target.
…hony targets The goal of this commit is to refactor the deployment makefiles, specifically focusing on transitioning from using phony targets to utilizing actual file targets. This change is implemented to enhance the declarative control over the configuration generation process, enabling more precise control over outcomes in the deployment management workflow. Changes: - Within the `config.mk` file located in `scripts/govtool/`, the adjustments were made to switch from initializing empty nginx files using the `touch` command to a more explicit and clear approach of creating empty files through `echo > $@`. This modification aligns with the objective of employing file targets instead of phony targets for better configuration generation control in the deployment process.
Refactoring the `prepare-dbsync-secrets` target within the deployment makefiles to improve configurability and usage of actual file targets. Changes - Reorganized the `prepare-config` target in the `config.mk` file to remove dependencies on phony targets and replaced them with actual file targets for more explicit configuration generation. - Separate file targets for `postgres_user`, `postgres_password`, and `postgres_db` within the `dbsync-secrets` directory were added to store the corresponding values securely. - Modified the rules within the `config.mk` file to utilize the new file targets instead of the previous phony target, enhancing the declarative control over the configuration generation process. This refactoring provides a clearer and more controlled approach to managing database synchronization secrets in the deployment process, ensuring that sensitive information is stored securely and efficiently within the designated files.
The purpose of this commit is to refactor the `generate-docker-compose-file` target within the `config.mk` file. The alteration involves replacing the existing use of the PHONY target with a file target, enabling a more file-target-driven approach for configuring the deployment process. Changes: - Updated the `prepare-config` target in `config.mk` to use the actual file target for `docker-compose.yml` instead of a PHONY target. This adjustment enhances the configuration control process by streamlining it to a file-target-centric method, ensuring better management and execution of the deployment tasks.
…et instead of a plain file target This commit aim to refactor the backend-config generation in the `config.mk` makefile by replacing the direct file target with a template file target approach. This adjustment enhances configurability and maintains consistency in handling configuration files across deployment setups. Changes: - Updated the `$(target_config_dir)/backend-config.json` target in the `config.mk` file to utilize `$(config_dir)/templates/backend-config.json.tpl` as a template, allowing for more structured and customizable configuration file generation.
The refactoring of deployment makefiles aims to transition from using phony targets to actual file targets. By replacing phony targets with file targets in the `config.mk` file, this commit enables more declarative control over the configuration generation process. The update ensures that only targets triggering specific actions remain as phony targets, maintaining functionality while improving clarity and control in deployment management. Changes: - Replaced phony target `prepare-config` dependencies with actual file targets, including `$(cardano_config_files)` and `$(target_config_dir)/cardano-node/config.json`, in the `config.mk` makefile. - Updated the `fetch-cardano-node-config` and `enable-prometheus` targets to create specific Cardano config files using actual file targets instead of phony targets. - Introduced the `cardano_configs` and `cardano_config_files` variables to manage the list of required Cardano config files more efficiently, leading to a more structured and easier-to-maintain deployment configuration setup.
…hony targets This commit transitions from using phony targets to using actual file targets in deployment makefiles. This change aims to provide more declarative control over the configuration generation process, enhancing predictability and maintainability. Changes: - Updated `config.mk` file in the `scripts/govtool` directory. - Replaced the phony target `prepare-config` dependencies with a new variable `output_files` that contains a list of required output files, improving clarity and maintainability of the configuration generation process. - Introduced the `outputs` variable to store a clear overview of all necessary files. - Modified the `prepare-config` target to utilize the `output_files` variable, ensuring that the generation of configuration files is now driven by actual targets rather than phony targets. - Enhanced the file-driven approach by declaring specific output files explicitly, maintaining phony targets only for tasks that involve executing particular actions. These changes facilitate a more structured and manageable configuration generation process by embracing file targets, improving the overall maintainability and predictability of the deployment configuration.
This commit modifies the `config.mk` makefile within the `scripts/govtool` directory by removing individual targets for Prometheus, Promtail, and Loki configuration files and replacing them with a single generic target. By altering the rules to utilize a single generic file target with pattern matching, the makefile becomes more concise and flexible, simplifying the maintenance of configuration file additions or changes. Changes: - Replaced individual targets for Prometheus, Promtail, and Loki configuration files with a generic rule using pattern matching in the `config.mk` makefile. - The new rule copies any `*.yml` file from the template directory to the target configuration directory, streamlining the process and enhancing maintainability. - The specific targets for Prometheus, Promtail, and Loki have been removed, reducing redundancy and increasing the overall clarity and efficiency of the configuration file handling.
Refactoring the Grafana provisioning target to enhance the deployment makefiles by replacing phony targets with actual file targets. This change improves the file-driven configuration generation process and allows for more declarative control over the configuration setup. Changes: - Modified the `config.mk` file within `scripts/govtool` directory. - Updated `config_subdirs` list to include additional Grafana provisioning subdirectories: datasources, dashboards, and alerting. - Updated `outputs` list to include files related to the newly added Grafana provisioning subdirectories. - Removed the `prepare-grafana-provisioning` as a phony target and directly integrated the actions within the respective file targets for datasources, dashboards, and alerting. - Added individual file targets for copying and processing datasources, dashboards, and the alerting configuration YAML files into the Grafana provisioning directory, ensuring customization capabilities. - Adjusted the sed command to modify the Grafana alerting configuration file path to the correct target directory path. This commit streamlines the Grafana provisioning process within the deployment makefiles, improving clarity, maintainability, and configurability of the deployment configurations.
This refactor aims to enhance the deployment makefiles by optimizing the `directories` variable usage for improved directory structuring. By ensuring better utilization of this variable, the file targets will be more accurately defined, allowing for a more organized and controlled configuration generation process. Changes made: - Updated the `config.mk` file in the `scripts/govtool` directory to reorganize the `directories` variable usage. - Introduced a new `cardano_config_provider` metadata variable for clarity and separation of concerns. - Restructured the variable assignments for `config_dir`, `template_config_dir`, `target_config_dir`, and `config_subdirs`. - Modified the file and directory definitions resulting in a more coherent structure and improved maintainability. - Removed duplicate assignments and enhanced directory creation commands using the `output_dirs` variable to streamline the process. - Adjusted dependencies and file generation rules to align with the updated directory structure, ensuring accurate target locations for file outputs. This refactor optimizes the directory variable usage in the `config.mk` file, improving the organization and maintainability of the deployment configuration generation process.
…server Implement a GitHub Action workflow file to automatically deploy the "test" branch changes to the corresponding test server. This workflow ensures that code changes pushed to the "test" branch trigger seamless deployment to the test environment, allowing for efficient testing processes. Changes: - Added a new workflow file "build-and-deploy-test.yml" under the ".github/workflows/" directory to facilitate the automatic deployment process for the "test" branch. - Configured the workflow to listen for changes pushed to the "test" branch exclusively, ensuring deployment only occurs for this specific branch. - Set up environment variables and deployment configurations specific to the test environment to enable successful deployment. - Included necessary steps within the workflow for checking out code, configuring AWS credentials, setting up SSH agent, deploying the application, and reprovisioning Grafana for effective deployment to the test server.
…ng server Implemented a GitHub Action workflow file to facilitate the automatic deployment of changes made to the "staging" branch to the corresponding staging server. This workflow ensures that code pushed to the "staging" branch triggers seamless deployment to the staging environment, streamlining the testing process. Changes: - Added a new workflow file named "build-and-deploy-staging.yml" under the ".github/workflows/" directory to automate the deployment process for the "staging" branch. - Configured the workflow to monitor and deploy changes pushed exclusively to the "staging" branch, ensuring deployment only occurs for this specific branch. - Set up environment variables and deployment configurations tailored to the staging environment to enable successful deployment. - Included necessary steps within the workflow for code checkout, AWS configuration, SSH agent setup, application deployment, and Grafana reprovisioning to facilitate effective deployment to the staging server.
…server Implemented a GitHub Action workflow file to automate the deployment of changes pushed to the "main" branch to the corresponding beta server. This workflow ensures that code alterations in the "main" branch trigger a smooth deployment process to the beta environment, facilitating effective testing and validation before production deployment. Changes: - Introduced a new workflow file named "build-and-deploy-beta.yml" within the ".github/workflows/" directory to automate the deployment process for the "main" branch. - Configured the workflow to exclusively monitor and deploy changes pushed to the "main" branch, ensuring deployment only occurs for this specific branch. - Defined specific environment variables and deployment configurations tailored to the beta server to ensure successful deployment. - Included necessary steps within the workflow for code checkout, AWS configuration, SSH agent setup, application deployment, and Grafana reprovisioning to enable seamless deployment to the beta server.
…e on selected server
Included a new GitHub Action workflow file,
"resync-cardano-node-and-db-sync.yml," to facilitate a manual
resynchronization of the Cardano Node and Cardano DB Sync on specific
target servers. This workflow allows for re-syncing the state of Cardano
to ensure accurate and up-to-date data for selected environments like
'dev', 'test', 'staging', or 'beta' servers, providing flexibility and
control over the synchronization process.
Changes:
- Added a new workflow file, "resync-cardano-node-and-db-sync.yml," in
the ".github/workflows/" directory to enable manual resynchronization
of the Cardano Node and Cardano DB Sync on target servers.
- Configured the workflow to be manually triggered with the option to
select the target environment ('dev', 'test', 'staging', or 'beta')
for the Cardano re-sync process.
- Set up environment variables and deployment configurations tailored to
the selected environment to ensure a successful re-sync operation.
- Included necessary steps within the workflow for checking out the
code, configuring AWS credentials, setting up SSH agent, destroying
existing Cardano Node, DB Sync, and Postgres database instances, and
deploying the application to perform the re-sync process effectively on
the chosen server.
The refactoring of deployment makefiles was necessary to transition from utilizing phony targets to actual file targets. This change was made to enhance the declarative control over the configuration generation process, streamlining the management of deployment configurations and improve readability of the outputs list.
The aim is to enhance the deployment makefiles by substituting phony targets with actual file targets. This adjustment allows for a more transparent control over the generation of configurations within the deployment setup. Changes: - Removed the invalid phony statement for $(target_config_dir)/cardano-node/config.json in the config.mk file of the govtool scripts. By eliminating the phony declaration, we are aligning with the goal of using proper file targets instead, which provides improved control over the configuration generation process.This adjustment ensures that the specific file target $(target_config_dir)/cardano-node/config.json is directly tied to its corresponding file and facilitates a more structured and predictable configuration generation mechanism in the deployment workflow.
By rearranging the order of targets in the `config.mk` file, this commit moves the `upload-config` target to a higher position within the file. This adjustment is made to enhance visibility and accessibility to the `upload-config` target, ensuring it is more prominent and easier to locate for users working on deployment configurations. Changes: - Moved the `upload-config` target to the top of the `config.mk` file to improve visibility and exposure. - No functionality changes were made; this adjustment aims to make the target more accessible within the file structure, enhancing developer workflow and understanding of the deployment makefiles.
…nt-makefiles-to-use-file-targets-instead-of-phony-targets [#436] Refactor deployment makefiles to use file targets instead of phony targets
To streamline the toggle maintenance workflow configuration, this commit simplifies the setup by removing the domain ENVs configuration and make arguments. These components are deemed unnecessary as make can now resolve domain variables and set default arguments from environment variables more efficiently. Changes: - Edited the workflow file: `.github/workflows/toggle-maintenance.yml` to adjust the name of the maintenance workflow and default values for maintenance mode. - Updated the Makefile in `scripts/govtool` by modifying the toggle-maintenance target to include docker-login and simplified the maintenance mode condition to use "enabled" instead of "enable" for consistency.
…e-maintenance-workflow-configuration [#444] Simplifying Toggle Maintenance workflow configuration
Modified the existing GitHub Action workflow file, renaming it from "build-and-deploy.yml" to "build-and-deploy-dev.yml," to limit manual deployment functionality specifically to the dev server. This adjustment ensures that the manual deploy option is available exclusively for the dev environment, enhancing control over deployment procedures for specific servers. Changes: - Renamed the workflow file from "build-and-deploy.yml" to "build-and-deploy-dev.yml" to designate manual deployment for the dev server solely. - Updated the workflow configuration to set the deployment environment to "dev" by default, restricting manual deployment to this environment. - Removed conditional logic related to setting the domain based on different environments to streamline the deployment process for the dev server. - Eliminated the step for destroying Cardano Node, DB sync, and Postgres instances as it was not necessary for manual deployments to the dev server.
fix sql error that could occur when there would be no voting power for predefined dreps Signed-off-by: jankun4 <michaljankun@gmail.com>
In order to enhance the ease of remote execution, the configuration files for PostgreSQL have been modified to rely on environment variable. This change enables smoother execution processes as it allows for flexible configurations that can adapt to different environments or setups. Changes: - Updated the `docker-compose.node+dbsync.yml` file to use environment variables for defining PostgreSQL configuration files, leveraging `$TARGET_HOST_CONFIG_DIR` for specifying file paths. This adjustment ensures that the execution setup is more versatile and can be easily adaptable for remote deployments.
In the context of the user story, which aimed to update the Cardano DB Sync version for Sanchonet to unblock tasks related to verification and adjustments, this commit bumps the version of Cardano DB Sync to 4.1.0 on a development stack providing only Cardano tools. By upgrading to the latest release, developers will be able to proceed with the necessary verifications and adjustments to ensure application compatibility with the updated DB Sync version. Changes: - Updated the `docker-compose.node+dbsync.yml` file to use the Cardano DB Sync image version 4.1.0 (`ghcr.io/intersectmbo/cardano-db-sync:sancho-4.1.0`).
Refactor deployment makefiles, automate GitHub Actions, improve configuration handling, fix SQL error, parametrize PostgreSQL files
In order to resolve deployment failures stemming from missing environment variables, the deployment workflows have been updated to incorporate the `IP_ADDRESS_BYPASSING_BASIC_AUTH1` and `IP_ADDRESS_BYPASSING_BASIC_AUTH2` variables. These variables are crucial for the application's deployment process, ensuring that all necessary environment variables are in place. By adding these variables to the deployment workflows, the application can now be deployed successfully without encountering errors related to missing essential environment variables. Changes: - Updated `.github/workflows/build-and-deploy-beta.yml`, `.github/workflows/build-and-deploy-dev.yml`, `.github/workflows/build-and-deploy-staging.yml`, and `.github/workflows/build-and-deploy-test.yml` to include `IP_ADDRESS_BYPASSING_BASIC_AUTH1` and `IP_ADDRESS_BYPASSING_BASIC_AUTH2` variables. These additions ensure that the required environment variables are present during deployment, preventing deployment failures due to missing variables.
…t-failures-due-to-missing-environment-variables [#516] Resolving deployment failures due to missing environment variables
…d-prettier-to-github-actions [#512] chore: add type checking & lint workflow to github actions
* add eslint config to govtool frontend package * add prettier config to unify the code style * parse frontend package codebase with the eslint --fix option * parse frontend package codebase with the prettier --fix option
…husky-lint-staged [#166] chore: add eslint & prettier configuration
…eanup chore: cleanup ga submission code
chore/update-CSL-to-19
Test candidate - fix
pmbinapps
requested review from
placek,
adgud,
kickloop,
MSzalowski,
Sworzen1 and
JanJaroszczak
as code owners
placek
approved these changes
Mar 25, 2024
MSzalowski
requested changes
Mar 25, 2024
| @@ -0,0 +1,78 @@ | |||
| name: Frontend Test, Lint & Type Check | |||
There was a problem hiding this comment.
This is already fixed on development. I would not push it to staging now
There was a problem hiding this comment.
OK, we can pull this first from dev to test, verify and then proceed to stage.
|
Could we wait for the #542 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
List of changes
Add:
[#484] Add GitHub Action workflow to automate deployment to the test stage
[#484] Add GitHub Action workflow to automate deployment to the staging stage
[#484] Add GitHub Action workflow to automate deployment to the beta stage
[#484] Add manually triggered workflow for resyncing the Cardano status
[#507] chore: add frontend test GitHub workflow
[#512] chore: add type checking & lint workflow to GitHub actions
[#512] chore: join test & code check workflows into one
[#512] chore: set .nvmrc as node version source
[#512] chore: split jobs to work in parallel; add emoji to label actions
[#166] chore: add eslint & prettier configuration
[#166] chore: enforce arrow component functions; disable obsolete or …
Fix:
[#501] fix SQL error
[#516] Update deployment workflows to include missing variables
chore: cleanup GA submission code
Change:
[#436] Change explicit file targets to a more generic target
[#457] Enable skipping Basic Auth for connections from VPN
[#457] Add VPN addresses as environment variables for Basic Auth bypass
[#512] chore: fix tsc execution
Remove:
[#436] Remove phony Cardano config targets with actual file targets
[#436] Remove the invalid phony statement
[#513] chore: cleanup GA submission code (assuming this is removing obsolete code)