feat: Add grafana Promtail to Kubernetes addon

[florentio]

What does this PR do?

Issue #774

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Yes, I have added a new example under examples to support my PR
• Yes, I have created another PR for add-ons under add-ons repo (if applicable)
• Yes, I have updated the docs for this feature
• Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs required examples and docs except a new pattern or add-on added.

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

[bryantbiggs]

Overall looks great - few minor comments.

Lets add an example for this - we'll call it grafana-loki - this is just the start where we have promtail and later we can add loki and connect the two.

You can copy this example and remove the EFS and Velero bits

[bryantbiggs]

This looks great, thank you!

[sondrelg]

I've just implemented this (thanks for adding it, it's super helpful; excited to see loki being added in the future), but I'm running into an issue where the promtail_helm_config values are not being picked up by terraform.

Is it possible that the loading of values from the promtail_helm_config object is broken?

This is my configuration (which I've tweaked several ways to try and get it to work, without luck):

  enable_promtail = true
  promtail_helm_config = {
    name       = "promtail"
    repository = "https://grafana.github.io/helm-charts"
    chart      = "promtail"
    version    = "6.3.0"
    namespace  = "promtail"
    values = [templatefile("${path.module}/promtail-values.yml", {})]
  }

My promtail-values.yml contains:

config:
  clients:
    - url: https://url-to-another-cluster.com/loki/api/v1/push

---

tl;dr: despite setting a URL value, the pods try to connect to the default value (http://loki-gateway/loki/api/v1/push) instead, and when I make changes to my config, the changes are not detected by terraform.

When I replace the above config with something nonsensical like aasdasd, the apply still runs fine. I'm pretty sure other addons would fail with a yaml-parsing error. This increases my suspicion that the contents of the file just aren't evaluated.

Is there a chance that something needs tweaking wrt. how the helm config is evaluated for this addon?

[sondrelg]

Let me know if you prefer I open an issue 🙏 Just thought it might be nice to mention it here first, since both of you might be able to answer relatively quickly.

[sondrelg]

For some added context, the config values are applied correctly when I use a normal helm release:

resource "helm_release" "loki" {
    name       = "promtail"
    repository = "https://grafana.github.io/helm-charts"
    chart      = "promtail"
    version    = "6.3.0"
    namespace  = "promtail"
    values = [templatefile("${path.module}/promtail-values.yml", {})]
}

Which also seems to point in the direction that something is wrong with addon loading (or my definition of it perhaps?)

[florentio]

Hello @sondrelg I couldn't reproduce your error, this is what I got where running terraform plan with your config

# module.eks.module.promtail.module.helm_addon.helm_release.addon[0] will be created
  + resource "helm_release" "addon" {
      + atomic                     = false
      + chart                      = "promtail"
      + cleanup_on_fail            = false
      + create_namespace           = true
      + dependency_update          = false
      + description                = "Promtail helm Chart deployment configuration"
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "promtail"
      + namespace                  = "promtail"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://grafana.github.io/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 1200
      + values                     = [
          + <<-EOT
                config:
                  clients:
                    - url: https://url-to-another-cluster.com/loki/api/v1/push
            EOT,
        ]
      + verify                     = false
      + version                    = "6.3.0"
      + wait                       = true
      + wait_for_jobs              = false

      + postrender {}
    }

and as you can see the pod is using https://url-to-another-cluster.com/loki/api/v1/push as url in the log file below
promtail-xq9xv.log

Can you open an issue and share the terraform plan section of resource "helm_release" "addon" so that we can see the plan content ?

[sondrelg]

Thanks for testing @florentio. I'll double check and report back in a bit 👍