[stable/prometheus-operator] 8.12.12 stuck in pending-install state

[Opalo]

Describe the bug
Installing prometheus-operator with the following piece of code:

resource "helm_release" "prometheus_operator" {
  depends_on = [
    module.eks.aws_eks_auth
  ]
  chart      = "stable/prometheus-operator"
  name       = "prometheus-operator"
  values     = [
    file("${path.module}/../../k8s/${local.environment}/prometheus-operator-values.yaml")
  ]
  wait       = false
  version    = "8.12.12"
}

ends with Error: Kubernetes cluster unreachable error and chart stuck with pending-install state.

Version of Helm and Kubernetes:

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

Which chart:
stable/prometheus-operator

What happened:
Chart stuck with pending-install status.

What you expected to happen:
Chart should be installed.

How to reproduce it (as minimally and precisely as possible):
No idea, to be honest - don't even know if this is reproducible.

Anything else we need to know:
When installing chart wit raw helm command: helm install prometheus-operator stable/prometheus-operator -f staging/prometheus-operator-values.yaml --version 8.12.12 everything works as expected. It takes about ~15 minutes for terraform script to crash with this unreachable error.

[forgems]

I can confirm. I've the same issue. In the debug log I've this

2020-04-15T17:23:15.947+0200 [DEBUG] plugin.terraform-provider-helm_v1.1.0_x4: 2020/04/15 17:23:15 [DEBUG] Starting delete for "prom-prometheus-operator-admission" Role                     
2020/04/15 17:23:19 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"                                                      
2020/04/15 17:23:19 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"                                                                                    
2020/04/15 17:23:19 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"                                       
2020-04-15T17:23:21.175+0200 [DEBUG] plugin.terraform-provider-helm_v1.1.0_x4: 2020/04/15 17:23:21 [DEBUG] Starting delete for "prom-prometheus-operator-admission" RoleBinding              
2020/04/15 17:23:24 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"                                                      
2020/04/15 17:23:24 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"                                                                                    
2020/04/15 17:23:24 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020-04-15T17:23:26.503+0200 [DEBUG] plugin.terraform-provider-helm_v1.1.0_x4: 2020/04/15 17:23:26 [DEBUG] Starting delete for "prom-prometheus-operator-admission" ServiceAccount
2020/04/15 17:23:29 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:29 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2020/04/15 17:23:29 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020-04-15T17:23:32.156+0200 [DEBUG] plugin.terraform-provider-helm_v1.1.0_x4: 2020/04/15 17:23:32 [DEBUG] Starting delete for "prom-prometheus-operator-admission-create" Job
2020-04-15T17:23:32.311+0200 [DEBUG] plugin.terraform-provider-helm_v1.1.0_x4: 2020/04/15 17:23:32 [DEBUG] creating 123 resource(s)
2020/04/15 17:23:34 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:34 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2020/04/15 17:23:34 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:39 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:39 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2020/04/15 17:23:39 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:44 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:44 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2020/04/15 17:23:44 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:49 [TRACE] dag/walk: vertex "provider.helm (close)" is waiting for "module.kubernetes_us_east.helm_release.prometheus"
2020/04/15 17:23:49 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"

After these lines it just repeats that it's waiting for helm_release. Stopped after 2 hours in pending state.
Which chart:

• stable/prometheus-operator
  Chart version
• 8.12.12
• 8.12.3

Kubernetes version

Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8+IKS", GitCommit:"c6910e70b932ed9ffe5baedade2274a72925660f", GitTreeState:"clean", BuildDate:"2020-03-13T14:13:41Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

terraform_provider_helm versions:

• 1.1.1
• 1.1.0

[Opalo]

Thanks @forgems! I'm somehow glad that you also have bumped into this. My log for terraform script execution can be found here.

[Opalo]

I also think it might be an issue with terraform helm provider rather with the chart itself.

[arlyon]

Just linking to this issue on the provider repo which seems to describe the same problem: hashicorp/terraform-provider-helm#467.

Tried it with 8.13.0, no luck.

[Opalo]

Tried it with 8.13.0, no luck.

Same here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.