Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Automate helm release with multiple images #2168

Closed
runningman84 opened this issue Jun 19, 2019 · 13 comments
Closed

Automate helm release with multiple images #2168

runningman84 opened this issue Jun 19, 2019 · 13 comments
Labels
flux2 Resolution suggested - already fixed in Flux v2 question

Comments

@runningman84
Copy link

Describe the bug
I would like to automate the prometheus operator helm release. My config looks like this:

---
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  name: prometheus-operator
  namespace: monitoring
  annotations:
    flux.weave.works/automated: 'true'
    flux.weave.works/tag.prometheus: semver:~2.10
    flux.weave.works/tag.alertmanager: semver:~0.16
    flux.weave.works/tag.grafana: semver:~6.2

Unfortunately nothing gets updated automatically. I would expect prometheus to update to 2.10 and grafana to 6.2.4

To Reproduce
Deploy prometheus operator with my sample config.

Expected behavior
Flux should use the newer images.

Logs


 $ fluxctl list-workloads -n monitoring
WORKLOAD                                                                     CONTAINER                   IMAGE                                              RELEASE   POLICY
monitoring:daemonset/prometheus-prometheus-node-exporter                     node-exporter               quay.io/prometheus/node-exporter:v0.17.0           ready     
monitoring:deployment/prometheus-grafana                                     grafana-sc-dashboard        kiwigrid/k8s-sidecar:0.0.16                        ready     
                                                                             grafana                     grafana/grafana:6.2.0                                        
                                                                             grafana-sc-datasources      kiwigrid/k8s-sidecar:0.0.16                                  
monitoring:deployment/prometheus-kube-state-metrics                          kube-state-metrics          k8s.gcr.io/kube-state-metrics:v1.5.0               ready     
monitoring:deployment/prometheus-prometheus-oper-operator                    prometheus-operator         quay.io/coreos/prometheus-operator:v0.29.0         ready     
monitoring:helmrelease/prometheus-operator                                                                                                                  DEPLOYED  automated
monitoring:statefulset/alertmanager-prometheus-prometheus-oper-alertmanager  alertmanager                quay.io/prometheus/alertmanager:v0.17.0            ready     
                                                                             config-reloader             quay.io/coreos/configmap-reload:v0.0.1                       
monitoring:statefulset/prometheus-prometheus-prometheus-oper-prometheus      prometheus                  quay.io/prometheus/prometheus:v2.9.1               ready     
                                                                             prometheus-config-reloader  quay.io/coreos/prometheus-config-reloader:v0.29.0            
                                                                             rules-configmap-reloader    quay.io/coreos/configmap-reload:v0.0.1                       
$ fluxctl list-images -n monitoring                                                                                                   
E0619 12:21:12.810749   14294 portforward.go:385] error copying from local connection to remote stream: read tcp4 127.0.0.1:39659->127.0.0.1:46076: read: connection reset by peer
WORKLOAD                                                                     CONTAINER                   IMAGE                                      CREATED
monitoring:daemonset/prometheus-prometheus-node-exporter                     node-exporter               quay.io/prometheus/node-exporter           
                                                                                                         |   master                                 12 Jun 19 18:59 UTC
                                                                                                         |   latest                                 04 Jun 19 16:51 UTC
                                                                                                         |   v0.18.1                                04 Jun 19 16:51 UTC
                                                                                                         |   v0.18.0                                09 May 19 23:23 UTC
                                                                                                         '-> v0.17.0                                30 Nov 18 15:59 UTC
                                                                                                             v0.17.0-rc.0                           20 Oct 18 06:36 UTC
                                                                                                             v0.16.0                                15 May 18 15:58 UTC
                                                                                                             v0.16.0-rc.3                           27 Apr 18 15:56 UTC
                                                                                                             v0.16.0-rc.2                           17 Apr 18 14:12 UTC
                                                                                                             v0.15.2                                05 Dec 17 14:55 UTC
monitoring:deployment/prometheus-grafana                                     grafana-sc-dashboard        kiwigrid/k8s-sidecar                       
                                                                                                         |   latest                                 11 Jun 19 11:20 UTC
                                                                                                         '-> 0.0.16                                 13 Mar 19 06:55 UTC
                                                                                                             0.0.15                                 13 Mar 19 06:47 UTC
                                                                                                             0.0.14                                 13 Mar 19 06:39 UTC
                                                                                                             0.0.13                                 12 Mar 19 07:47 UTC
                                                                                                             0.0.12                                 04 Mar 19 06:36 UTC
                                                                                                             0.0.11                                 19 Feb 19 07:39 UTC
                                                                                                             0.0.10                                 14 Feb 19 07:23 UTC
                                                                                                             0.0.9                                  14 Feb 19 07:20 UTC
                                                                                                             0.0.8                                  14 Feb 19 07:16 UTC
                                                                             grafana                     grafana/grafana                            
                                                                                                         |   master                                 18 Jun 19 12:52 UTC
                                                                                                         |   6.2.4                                  18 Jun 19 11:10 UTC
                                                                                                         |   6.2.2                                  05 Jun 19 13:25 UTC
                                                                                                         |   latest                                 05 Jun 19 13:25 UTC
                                                                                                         |   6.2.1                                  27 May 19 13:08 UTC
                                                                                                         '-> 6.2.0                                  22 May 19 11:34 UTC
                                                                                                             6.2.0-beta2                            15 May 19 11:49 UTC
                                                                                                             6.2.0-beta1                            07 May 19 14:33 UTC
                                                                                                             6.1.6                                  29 Apr 19 14:04 UTC
                                                                                                             5.4.4                                  29 Apr 19 12:18 UTC
                                                                             grafana-sc-datasources      kiwigrid/k8s-sidecar                       
                                                                                                         |   latest                                 11 Jun 19 11:20 UTC
                                                                                                         '-> 0.0.16                                 13 Mar 19 06:55 UTC
                                                                                                             0.0.15                                 13 Mar 19 06:47 UTC
                                                                                                             0.0.14                                 13 Mar 19 06:39 UTC
                                                                                                             0.0.13                                 12 Mar 19 07:47 UTC
                                                                                                             0.0.12                                 04 Mar 19 06:36 UTC
                                                                                                             0.0.11                                 19 Feb 19 07:39 UTC
                                                                                                             0.0.10                                 14 Feb 19 07:23 UTC
                                                                                                             0.0.9                                  14 Feb 19 07:20 UTC
                                                                                                             0.0.8                                  14 Feb 19 07:16 UTC
monitoring:deployment/prometheus-kube-state-metrics                          kube-state-metrics          k8s.gcr.io/kube-state-metrics              image data not available
                                                                                                         '-> v1.5.0                                 ?
monitoring:deployment/prometheus-prometheus-oper-operator                    prometheus-operator         quay.io/coreos/prometheus-operator         
                                                                                                         |   master                                 07 Jun 19 12:19 UTC
                                                                                                         |   v0.30.1                                31 May 19 20:47 UTC
                                                                                                         |   v0.30.0                                10 May 19 20:42 UTC
                                                                                                         '-> v0.29.0                                20 Feb 19 11:11 UTC
                                                                                                             v0.28.0                                29 Jan 19 14:45 UTC
                                                                                                             v0.27.0                                08 Jan 19 16:25 UTC
                                                                                                             v0.26.0                                30 Nov 18 12:55 UTC
                                                                                                             v0.25.0                                24 Oct 18 13:25 UTC
                                                                                                             v0.24.0                                11 Oct 18 12:05 UTC
                                                                                                             v0.23.2                                23 Aug 18 13:36 UTC
monitoring:helmrelease/prometheus-operator                                                                                                          
monitoring:statefulset/alertmanager-prometheus-prometheus-oper-alertmanager  alertmanager                quay.io/prometheus/alertmanager            
                                                                                                         |   master                                 14 Jun 19 13:17 UTC
                                                                                                         |   latest                                 03 May 19 09:48 UTC
                                                                                                         '-> v0.17.0                                03 May 19 09:48 UTC
                                                                                                             v0.16.2                                05 Apr 19 12:56 UTC
                                                                                                             v0.16.1                                31 Jan 19 15:34 UTC
                                                                                                             v0.16.0                                21 Jan 19 13:25 UTC
                                                                                                             v0.16.0-beta.0                         15 Jan 19 20:22 UTC
                                                                                                             v0.16.0-alpha.0                        30 Nov 18 22:41 UTC
                                                                                                             v0.15.3                                09 Nov 18 16:02 UTC
                                                                                                             v0.15.2                                14 Aug 18 11:13 UTC
                                                                             config-reloader             quay.io/coreos/configmap-reload            
                                                                                                         |   latest                                 02 Jan 17 23:21 UTC
                                                                                                         '-> v0.0.1                                 02 Jan 17 23:21 UTC
monitoring:statefulset/prometheus-prometheus-prometheus-oper-prometheus      prometheus                  quay.io/prometheus/prometheus              
                                                                                                         |   master                                 17 Jun 19 19:40 UTC
                                                                                                         |   latest                                 25 May 19 13:59 UTC
                                                                                                         |   v2.10.0                                25 May 19 13:59 UTC
                                                                                                         |   v2.10.0-rc.0                           22 May 19 13:52 UTC
                                                                                                         |   v2.9.2                                 24 Apr 19 16:59 UTC
                                                                                                         '-> v2.9.1                                 16 Apr 19 19:18 UTC
                                                                                                             v2.9.0                                 15 Apr 19 10:38 UTC
                                                                                                             v2.9.0-rc.0                            10 Apr 19 21:50 UTC
                                                                                                             v2.8.1                                 28 Mar 19 19:21 UTC
                                                                                                             v2.8.0                                 12 Mar 19 09:05 UTC
                                                                             prometheus-config-reloader  quay.io/coreos/prometheus-config-reloader  
                                                                                                         |   master                                 07 Jun 19 12:19 UTC
                                                                                                         |   v0.30.1                                31 May 19 20:47 UTC
                                                                                                         |   v0.30.0                                10 May 19 20:42 UTC
                                                                                                         '-> v0.29.0                                20 Feb 19 11:11 UTC
                                                                                                             v0.28.0                                29 Jan 19 14:46 UTC
                                                                                                             v0.27.0                                08 Jan 19 16:25 UTC
                                                                                                             v0.26.0                                30 Nov 18 12:55 UTC
                                                                                                             v0.25.0                                24 Oct 18 13:25 UTC
                                                                                                             v0.24.0                                11 Oct 18 12:05 UTC
                                                                                                             v0.23.2                                23 Aug 18 13:36 UTC
                                                                             rules-configmap-reloader    quay.io/coreos/configmap-reload            
                                                                                                         |   latest                                 02 Jan 17 23:21 UTC
                                                                                                         '-> v0.0.1                                 02 Jan 17 23:21 UTC

Additional context
Add any other context about the problem here, e.g

  • Flux version: 1.13.0
  • Helm Operator version: 0.9.2
  • Kubernetes version: 1.12
  • Git provider: github
  • Container registry provider: dockerhub / other
@runningman84 runningman84 added blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels Jun 19, 2019
@hiddeco
Copy link
Member

hiddeco commented Jun 19, 2019

We need a (redacted) copy of your HelmRelease (including the values field) to confirm if this is a bug.

@runningman84
Copy link
Author

Does this help?

$ kubectl describe helmrelease prometheus-operator -n monitoring                                                                      
Name:         prometheus-operator
Namespace:    monitoring
Labels:       flux.weave.works/sync-gc-mark=sha256.WmJXjMVidRbo08dV5CYwOf5Wa00UQECRwnK1FChHsRo
Annotations:  flux.weave.works/automated: true
              flux.weave.works/sync-checksum: c00e156ad15c82f240fc52be222147c4d58e42c3
              flux.weave.works/tag.alertmanager: semver:~0.16
              flux.weave.works/tag.grafana: semver:~6.2
              flux.weave.works/tag.prometheus: semver:~2.10
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"flux.weave.works/v1beta1","kind":"HelmRelease","metadata":{"annotations":{"flux.weave.works/automated":"true","flux.weave.w...
API Version:  flux.weave.works/v1beta1
Kind:         HelmRelease
Metadata:
  Creation Timestamp:  2019-06-18T15:06:19Z
  Generation:          4
  Resource Version:    14404959
  Self Link:           /apis/flux.weave.works/v1beta1/namespaces/monitoring/helmreleases/prometheus-operator
  UID:                 a0bbad80-91da-11e9-96c8-02a228bf1dc8
Spec:
  Chart:
    Name:        prometheus-operator
    Repository:  https://kubernetes-charts.storage.googleapis.com/
    Version:     5.10.5
  Release Name:  prometheus
  Values:
    Alertmanager:
      Enabled:  true

@hiddeco
Copy link
Member

hiddeco commented Jun 19, 2019

You need to specify the images within the .spec.values for Flux to be able to update the image refs.

@hiddeco hiddeco added question and removed blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels Jun 19, 2019
@runningman84
Copy link
Author

ah I thought the default values would be used from the chart... manually specifying them worked for me.

@hiddeco
Copy link
Member

hiddeco commented Jun 19, 2019

We recently added a note to the documentation about why this is required: 1c1bf10

@runningman84
Copy link
Author

How does it work with the traefik chart, it uses the paramater imageTag: 1.7.12
What would be the correct annotation?

@hiddeco
Copy link
Member

hiddeco commented Jun 19, 2019

This does not work yet, I am working on a feature which makes it possible to configure YAML dot notation paths for such cases, but before I am able to land this the separation for registry keys (#2149) needs to be in place first.

@runningman84
Copy link
Author

What about updating helm charts instead of images, are you going to implement that too? This would make updating helm applications quite easy.

@hiddeco
Copy link
Member

hiddeco commented Jun 19, 2019

This is already semi-supported by using charts from git sources, we have no plans to implement this for 'published' charts from repositories, as we can not guarantee those charts do not contain any breaking changes that would corrupt the (existing) release in a non-revertible way.

@runningman84
Copy link
Author

yes I already use it for git based charts. I would be great to have a way to use this also for published charts. If you could configure only minor updates I think this would not be a big deal.

I also miss some info how to handle cases where only the latest tag is published. Is flux able to recognize that the image behind the latest tag changed? Which would be the correct annotation?

@stefanprodan
Copy link
Member

Flux does not support latest tag or any other mutable tag.

@tjanson
Copy link

tjanson commented Sep 18, 2019

as we can not guarantee those charts do not contain any breaking changes that would corrupt the (existing) release in a non-revertible way

@hiddeco Just out of curiosity: The Charts pulled from git repos can contain arbitrary changes, and that’s (unintentionally?) supported? While automated updates from Helm repos are ruled out for that same reason?
Please don’t take this as backhanded criticism – I just genuinely don’t understand. Personally, I’d love to be able to automate Helm Chart upgrades.

@kingdonb kingdonb added the flux2 Resolution suggested - already fixed in Flux v2 label Jul 28, 2021
@kingdonb
Copy link
Member

Flux v2 supports image automation in HelmReleases, as well as automatic release upgrades.

Please check out the latest developments in Flux v2!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
flux2 Resolution suggested - already fixed in Flux v2 question
Projects
None yet
Development

No branches or pull requests

5 participants