Grafana cloud to IOG ec2 monitoring

[johnalotoski]

Overview:

• Migrate from grafana cloud monitoring to ec2 monitoring, add resource tagging support, declarative route53 CNAME list, and additional improvements and fixes.

Details:

• Bumps capkgs for compose-process, node updates
• Fixes kes rotation job and adjusts perms on new kes rotation files
• Improves filtering unwanted varnish traffic to reduce cache miss metrics
• Adds cluster flakeModule options for generic infra tagging
• Adds a systemd bootstrap key removal service after a 1 week delay in preference of auth-keys-hub ssh key mgmt
• Adds opsLib to nixos cardano-parts perNode module access
• Update the node mainnet snapshot url from s3 to CF now that the artifacts support range requests
• Template updates:
  • Adds staging/prod book handling, and set gdb auto-load to no for justfile recipe
  • Accepts TF per node ami spec
  • Switches from grafana cloud to iog ec2 monitoring
  • Updates existing dashboards for a generic grafana prom datasource
  • Adds a cardano-monitoring blackbox integration alert
  • Adds more dashs, mod for generic prom ds and tf escaping
  • Adds a mimir alertmanager bootstrap recipe
  • Makes cluster cname records declarative

Breaking changes:

• Tofu Grafana change from cloud to iog monitoring will require:
  • Updating secrets in secrets/monitoring/* and secrets/tf/grafana.tfvars
  • Updating flake/terraform/grafana.nix from the updated template:
    • templates/cardano-parts-project/flake/terraform/grafana.nix
  • Adding new blackbox alert and endpoint declaration:
    • templates/cardano-parts-project/flake/terraform/grafana/alerts/blackbox.nix-import
    • templates/cardano-parts-project/flake/terraform/grafana/blackbox/blackbox.nix-import
  • Updating your infra.grafana.stackName at flake/cluster.nix
• Tofu cluster resource declarative CNAMEs will require:
  • Updating flake/terraform/cluster.nix from the updated template:
    • templates/cardano-parts-project/flake/terraform/cluster.nix
  • Declaring a route53 custom resource file at flake/terraform/cluster/route53.nix-import from the updated template:
    • templates/cardano-parts-project/flake/terraform/cluster/route53.nix-import
• Generic infra resource tagging will require:
  • Declaration of flake.cardano-parts.cluster.infra.generic.{organization,tribe,function,repo} attributes in your consuming repos flake/cluster.nix. See the following template file for an example:
    • templates/cardano-parts-project/flake/cluster.nix