Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: add Argocd application autodiscovery #1911

Closed
cloudoutloud opened this issue Feb 1, 2024 · 30 comments · Fixed by #2158 or #2182
Closed

Feature Request: add Argocd application autodiscovery #1911

cloudoutloud opened this issue Feb 1, 2024 · 30 comments · Fixed by #2158 or #2182
Assignees
Milestone

Comments

@cloudoutloud
Copy link
Contributor

Is your feature request related to a problem?

I would like to automatically update the helm charts managed by the Argocd application manifest a repoURL sets the upstream chart and the targetRevision sets the version number.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal

Solution you'd like

I would like it to look for all files in a repo.

Alternatives you've considered

No response

Anything else?

No response

@olblak
Copy link
Member

olblak commented Feb 5, 2024

Hey,
It would definitely makes.
This could be new autodiscovery plugin named argocd

The goal of a autodiscovery is to parse files to gather all information needed to generate Updatecli manifest

The Rancher fleet plugin could probably be used as a source of inspiration

Then that new plugin would need to be mapped to Updatecli on the following places

Notes that fleet relies on file named fleet.yaml or fleet.yml, in the context of argocd this could be any yaml files with the field apiVersion set to argoproj.io

@cloudoutloud
Copy link
Contributor Author

@olblak I can work on this how can I assign myself?

@olblak
Copy link
Member

olblak commented Feb 17, 2024

@olblak I can work on this how can I assign myself?

More than happy to assign it to you.
Feel free to reach out if you need some assistance.
Also we are a few to hang on Matrix to chat if you want to join us

https://app.gitter.im/#/room/#Updatecli_community:gitter.im

@olblak
Copy link
Member

olblak commented Feb 24, 2024

@cloudoutloud have you started this feature yet?
I am almost done with flux support and I could implemented the argocd support next while I am in this area of the code :)

@olblak
Copy link
Member

olblak commented Feb 24, 2024

@cloudoutloud have you started this feature yet? I am almost done with flux support and I could implemented the argocd support next while I am in this area of the code :)

Otherwise I could pick up something else as there are plenty stuff to improve anyway

@cloudoutloud
Copy link
Contributor Author

@olblak
I briefly looked at what was need for the change.
Planning to look at week commencing 4th March as away on holiday now 😀

@olblak
Copy link
Member

olblak commented Feb 24, 2024

Awesome thanks, then I'll pick something else, enjoy your vacation :)

@paltaa
Copy link

paltaa commented Apr 12, 2024

Any updates on this? Integration with ArgoCD or a plugin would be greatly appreciated

@olblak
Copy link
Member

olblak commented Apr 15, 2024

I think @cloudoutloud wanted to look into this but I am fine picking up here.

@cloudoutloud
Copy link
Contributor Author

@olblak
Apologies feel free to pick this up, I have not found the time.
I don't want to stall the implementation.

@olblak
Copy link
Member

olblak commented Apr 24, 2024

Apologies feel free to pick this up, I have not found the time.

No need to apologies, don't worry that's totally fine.
I'll try to find some time to implement the support

@olblak
Copy link
Member

olblak commented May 8, 2024

I opened an pullrequest and I am looking for a real argocd git repository containing applications to update :)

@olblak
Copy link
Member

olblak commented May 9, 2024

@cloudoutloud @paltaa

My pullrequest only handles helm version update.
Feel free to share if other kind of version could be updated in a argocd manifest

#######################
# LOCAL AUTODISCOVERY #
#######################


###########################################################################################################
# DEPS(HELM): BUMP HELM CHART "SEALED-SECRETS" IN ARGOCD MANIFEST "TESTDATA/SEALED-SECRETS/MANIFEST.YAML" #
###########################################################################################################


SOURCES
=======
WARNING: Updatecli binary version is unset. This means you are using a development version that ignores manifest version constraint.

sealed-secrets
--------------
Searching for version matching pattern "*"
✔ Helm Chart "sealed-secrets" version "2.15.3" is found from repository "https://bitnami-labs.github.io/sealed-secrets"

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: sealed-secrets
Helm chart for the sealed-secrets controller.
Project Home: https://github.com/bitnami-labs/sealed-secrets
Require Kubernetes Version: >=1.16.0-0
Version created on the 2024-04-04 16:40:40.720237525 +0000 UTC

Sources:

* https://github.com/bitnami-labs/sealed-secrets



URL:

* https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.15.3/sealed-secrets-2.15.3.tgz




CHANGELOG:
----------

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: sealed-secrets
Helm chart for the sealed-secrets controller.
Project Home: https://github.com/bitnami-labs/sealed-secrets
Require Kubernetes Version: >=1.16.0-0
Version created on the 2024-04-04 16:40:40.720237525 +0000 UTC

Sources:

* https://github.com/bitnami-labs/sealed-secrets



URL:

* https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.15.3/sealed-secrets-2.15.3.tgz





CONDITIONS:
===========

sealed-secrets-name
-------------------
✔ key "$.spec.source.chart" is correctly set to "sealed-secrets"

sealed-secrets-repository
-------------------------
✔ key "$.spec.source.repoURL" is correctly set to "https://bitnami-labs.github.io/sealed-secrets"


TARGETS
========

sealed-secrets
--------------

**Dry Run enabled**

⚠ - change detected:
	* key "$.spec.source.targetRevision" should be updated from "1.16.1" to "2.15.3", in file "testdata/sealed-secrets/manifest.yaml"


ACTIONS
========


=============================

REPORTS:



✔ Local AutoDiscovery:

⚠ deps(helm): bump Helm chart "sealed-secrets" in ArgoCD manifest "testdata/sealed-secrets/manifest.yaml":
	Source:
		✔ [sealed-secrets] Get latest "sealed-secrets" Helm chart version
	Condition:
		✔ [sealed-secrets-name] Ensure Helm chart name sealed-secrets is specified
		✔ [sealed-secrets-repository] Ensure Helm chart repository https://bitnami-labs.github.io/sealed-secrets is specified
	Target:
		⚠ [sealed-secrets] deps(helm): bump Helm chart "sealed-secrets" in ArgoCD manifest "testdata/sealed-secrets/manifest.yaml"


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	0
  * Succeeded:	1
  * Total:	2

@paltaa
Copy link

paltaa commented May 9, 2024

Hey @olblak, is there a possibility to specify the keys to upgrade? right now im using multiple sources for some of the apps, also if you got any instructions to test your PR i would be happy to do so,

Cheers

@olblak
Copy link
Member

olblak commented May 11, 2024

if you got any instructions to test your PR i would be happy to do so,

@paltaa you can test the pullrequest by running

# Build a new Updatecli binary from the root directory of my PR
go build -o bin/updatecli .
# Execute Updatecli from the root directory of you argocd project
./bin/updatecli diff
# To see generated manifest
./bin/updatecli manifest show

is there a possibility to specify the keys to upgrade

do you have more keys than targetRevision to update?

@olblak olblak added this to the 0.77.0 milestone May 11, 2024
@olblak
Copy link
Member

olblak commented May 13, 2024

@paltaa I just released argocd support in https://github.com/updatecli/updatecli/releases/tag/v0.77.0 with docs on https://www.updatecli.io/docs/plugins/autodiscovery/argocd/

@paltaa
Copy link

paltaa commented May 13, 2024

Hey @olblak i've been testing the plugin with autodiscovery but the PRs are not being created and im not sure why, cant see anything that could point me to the right direction in the logs with debug:

➜  gitops git:(main) ✗ updatecli diff --debug --values updatecli.d/cfg.yaml


+++++++++++
+ PREPARE +
+++++++++++

DEBUG: Default Updatecli manifest directory detected "updatecli.d"
Loading Pipeline "updatecli.d/cfg.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: pipeline name undefined, we'll use the manifest file checksum
Loading Pipeline "updatecli.d/default.yaml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid

SCM repository retrieved: 1
DEBUG: cloning git repository: https://github.com/Harbor-Systems/gitops.git in /var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops
DEBUG: cloning git repository: https://github.com/Harbor-Systems/gitops.git in /var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops
DEBUG: repository already exists, trying to pull changes
DEBUG: Fetching remote branches for resetting local ones


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++++++++++++++++++++++
+ ARGOCD AUTODISCOVERY EXAMPLE +
++++++++++++++++++++++++++++++++

DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"


ARGOCD
=======
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: 2 Argocd manifest(s) found
DEBUG:     * "helm/dev/prometheus-operator-crds/application.yaml"
DEBUG:     * "helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: parsing file "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: file path "helm/dev/prometheus-operator-crds/application.yaml" matching rule "helm/dev/*/application.yaml"
DEBUG: parsing file "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: ArgoCD manifest: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/prod/prometheus-operator-crds/application.yaml"
DEBUG: Helm Chart Name: "prometheus-operator-crds"
DEBUG: Helm Repository URL: "https://prometheus-community.github.io/helm-charts"
DEBUG: Version: "7.0.0"
DEBUG: Ignoring Helm chart "prometheus-operator-crds" from "helm/prod/prometheus-operator-crds/application.yaml", as not matching only rule(s)
Manifest detected: 1


---

=> Total manifest detected: 1



++++++++++++
+ PIPELINE +
++++++++++++



############
# CFG.YAML #
############


################################
# ARGOCD AUTODISCOVERY EXAMPLE #
################################


##################################################################################################################################
# DEPS(HELM): BUMP HELM CHART "PROMETHEUS-OPERATOR-CRDS" IN ARGOCD MANIFEST "HELM/DEV/PROMETHEUS-OPERATOR-CRDS/APPLICATION.YAML" #
##################################################################################################################################


SOURCES
=======

prometheus-operator-crds
------------------------
Searching for version matching pattern "*"
✔ Helm Chart "prometheus-operator-crds" version "11.0.0" is found from repository "https://prometheus-community.github.io/helm-charts"

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: prometheus-operator-crds
A Helm chart that collects custom resource definitions (CRDs) from the Prometheus Operator, allowing for seamless integration with GitOps tools

Require Kubernetes Version: >=1.16.0-0
Version created on the 2024-04-06 20:25:10.906667449 +0000 UTC

Sources:

* https://github.com/prometheus-community/helm-charts



URL:

* https://github.com/prometheus-community/helm-charts/releases/download/prometheus-operator-crds-11.0.0/prometheus-operator-crds-11.0.0.tgz




CHANGELOG:
----------

Remark: We couldn't identify a way to automatically retrieve changelog information.
Please use following information to take informed decision

Helm Chart: prometheus-operator-crds
A Helm chart that collects custom resource definitions (CRDs) from the Prometheus Operator, allowing for seamless integration with GitOps tools

Require Kubernetes Version: >=1.16.0-0
Version created on the 2024-04-06 20:25:10.906667449 +0000 UTC

Sources:

* https://github.com/prometheus-community/helm-charts



URL:

* https://github.com/prometheus-community/helm-charts/releases/download/prometheus-operator-crds-11.0.0/prometheus-operator-crds-11.0.0.tgz





CONDITIONS:
===========

prometheus-operator-crds-repository
-----------------------------------
DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
✔ key "$.spec.source.repoURL" is correctly set to "https://prometheus-community.github.io/helm-charts"

prometheus-operator-crds-name
-----------------------------
DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
✔ key "$.spec.source.chart" is correctly set to "prometheus-operator-crds"


TARGETS
========

prometheus-operator-crds
------------------------

**Dry Run enabled**

DEBUG: checkout git branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e", based on "main"
DEBUG: Checking if branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" diverged from "main":
DEBUG: 	all good,branch "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e" is ahead of "main"
DEBUG: Relative path detected: changing from "helm/dev/prometheus-operator-crds/application.yaml" to absolute path from SCM: "/var/folders/kp/shczqhd56kx8td73p3gzl5j40000gn/T/updatecli/github/Harbor-Systems/gitops/helm/dev/prometheus-operator-crds/application.yaml"
⚠ - change detected:
	* key "$.spec.source.targetRevision" should be updated from "'7.0.0'" to "11.0.0", in file "helm/dev/prometheus-operator-crds/application.yaml"
DEBUG: Checking if local changes have been done that should be published
DEBUG: no changes detected between branches "main" and "updatecli_main_26212ee07681e41117e1615c809e501e3b20d6236dc8113ae90444c103f0f84e"


ACTIONS
========


=============================

REPORTS:



✔ Argocd Autodiscovery Example:

- CFG.YAML:

⚠ deps(helm): bump Helm chart "prometheus-operator-crds" in ArgoCD manifest "helm/dev/prometheus-operator-crds/application.yaml":
	Source:
		✔ [prometheus-operator-crds] Get latest "prometheus-operator-crds" Helm chart version
	Condition:
		✔ [prometheus-operator-crds-name] Ensure Helm chart name prometheus-operator-crds is specified
		✔ [prometheus-operator-crds-repository] Ensure Helm chart repository https://prometheus-community.github.io/helm-charts is specified
	Target:
		⚠ [prometheus-operator-crds] deps(helm): bump Helm chart "prometheus-operator-crds" in ArgoCD manifest "helm/dev/prometheus-operator-crds/application.yaml"


Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	0
  * Skipped:	1
  * Succeeded:	1
  * Total:	3

Also what i ment by specifying the keys to update is that in ArgoCD applications you can use multiple sources for the same application, for example (just the sources):

  sources:
    - chart: emqx
      repoURL: 'https://repos.emqx.io/charts'
      targetRevision: '5.5.1'
      helm:
        valueFiles:
          - $values/helm/dev/emqx/values.yaml
    - repoURL: 'redacted
      targetRevision: HEAD
      ref: values

In this case i would like to update targetRevision only for chart: emqx. This is the way its setup so you can get values from another or the same repository using gitops for external charts.

Is this supported? Any hints on why the PRs are not created?

Cheers

@olblak
Copy link
Member

olblak commented May 13, 2024

Mmmh I wasn't aware of sources, I just implemented source as described in this issue
I am reopening this issue

@olblak olblak reopened this May 13, 2024
@paltaa
Copy link

paltaa commented May 13, 2024

@olblak Thanks! Let me know when you have something to test

@olblak
Copy link
Member

olblak commented May 14, 2024

@paltaa Another pullrequest that you can test.
All you need is building a new binary running go build -o bin/updatecli . and then use it on your repo

@paltaa
Copy link

paltaa commented May 15, 2024

@olblak Perfect! Thanks man, will test it right now and comment back

@paltaa
Copy link

paltaa commented May 15, 2024

Alright!

Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	3
  * Skipped:	27
  * Succeeded:	23
  * Total:	54

Charts that failed are hosted in private repositories so this should be expected, thanks a lot man!

The only doubt that i have now is why the PRs are not being created? am i missing any config? this is the config i have right now:

updatecli.d/default.yaml

name: Argocd Autodiscovery Example

autodiscovery:
  # scmid is applied to all crawlers
  scmid: default
  crawlers:
    argocd:
      # To ignore specific path
      ignore:
        # Ignore manifest match path "dev/*.yaml"
        - path: helm/prod/*.yaml
        # Ignore any Helm chart named "kubewarden-crds"
        - charts:
            "kubewarden-crds": ""
        # Ignore any helm chart repository named "https://charts.jetstack.io"
        - repositories:
            - "https://charts.jetstack.io"

        # # Ignore any Helm chart named "kubewarden-crds" in manifest path "dev/*.yaml"
        # - path: dev/*.yaml
        #   charts:
        #     "kubewarden-crds": ""
      # Only accepts the same rule than ignore
      only:
       - path: helm/dev/*/application.yaml
      #  - charts:
      #      "kubewarden-crds": ""
      #  - repositories:
      #      - "https://charts.jetstack.io"
      
scms:
  default:
    kind: github
    spec:
      #user: redacted
      owner: redacted
      username: redacted
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      repository: gitops
      branch: "main"
    
actions:
    default:
        kind: "github/pullrequest"
        spec:
            labels:
                - "dependencies"
            mergemethod: "squash"
        scmid: "default"

Also, if i run an apply, the only changes commited to the new branch just have 1 source, only one case for our gitops repo

@olblak
Copy link
Member

olblak commented May 16, 2024

The only doubt that i have now is why the PRs are not being created?

You are missing the actionid in the autodiscovery

autodiscovery:
  # scmid is applied to all crawlers
  scmid: default

should become

autodiscovery:
  actionid: default
  # scmid is applied to all crawlers
  scmid: default

@olblak
Copy link
Member

olblak commented May 16, 2024

Also, if i run an apply, the only changes commited to the new branch just have 1 source, only one case for our gitops repo

I am wondering why you have some many skipped pipelines according your run summary

Run Summary
===========
Pipeline(s) run:
  * Changed:	1
  * Failed:	3
  * Skipped:	27
  * Succeeded:	23
  * Total:	54

@paltaa
Copy link

paltaa commented May 16, 2024

Hey @olblak, added your change and the PR is created but still just commits the change for my one chart with one source, any other config needed?

Ok, it does not like repoURLs with ' '. After removing them the commits are being done! so thanks man.
✗ key "$.spec.sources[0].repoURL", is incorrectly set to "'https://victoriametrics.github.io/helm-charts/'" and should be "https://victoriametrics.github.io/helm-charts/"

Also, all of the skipped is the production application that i dont want to automatically update, so that is working fine

EDIT: Everything works now

Just one last thing, only if possible, can we have 1 pr per dependency? instead of 1 pr for all the upgrades? hehe

@olblak
Copy link
Member

olblak commented May 17, 2024

@paltaa

Ok, it does not like repoURLs with ' '. After removing them the commits are being done! so thanks man.

Do you think it it's an issue on Updatecli? Was your manifest initially working? Because I would imagine argocd to fail as well

Just one last thing, only if possible, can we have 1 pr per dependency? instead of 1 pr for all the upgrades? hehe

Yes can set groupby to either individual or all where individual opens one pull request per dependency while all group them all like here

Also combined with versionfilter you could group all patch update together
I realise now the indentation for versionfilter documentation is broken on https://www.updatecli.io/docs/plugins/autodiscovery/argocd/

@paltaa
Copy link

paltaa commented May 17, 2024

Great! Thanks for all the help man, really appreciate it.

'' were working with ArgoCD so i would think its just an updatecli issue

@paltaa
Copy link

paltaa commented May 27, 2024

Hey @olblak any date for the release which will contain this new changes?

@olblak
Copy link
Member

olblak commented May 27, 2024

Hey @olblak any date for the release which will contain this new changes?

What about now?
Give a moment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
3 participants