Automate Helm Chart Update

[olblak]

Automate Helm Chart Update

Fix #761

• Without manifest
• With manifest
• SCM Support
• Pullrequest support
• Documentation
• Dockerimage should be using semver
• Values.yaml must be in the same directory than the chart.yaml

This pullrequest is an attempt to automatically generate Updatecli manifest based on a local directory.
I envision two different scenarios, with and without manifest

Without

The default scenario, if we run updatecli diff or updatecli apply, by default we try to detect available updates from the local directory. Until #716 is done, the default scenario won't support scm

With manifest

This scenario relies on a new configuration, to customize the way we build manifest.
We use this configuration to provide scm configuration, or for example restrict generated manifest based on rules.

We can run one of the following command

• updatecli manifest show --config example.yaml
• updatecli diff --config example.yaml
• updatecli apply --config example.yaml

.example.yaml

# **Optional** Provide generic scm configuration used to generate and/or apply Updatecli manifest
# If an scm configuration is provided then we use that repository to automatically generate manifest 
scms:
  test:
    kind: github
    spec:
      owner: epinio
      repository: helm-charts
      token: '{{ requiredEnv "UPDATECLI_GITHUB_TOKEN" }}'
      username: '{{ requiredEnv "UPDATECLI_GITHUB_ACTOR" }}'
      branch: main

# Provide generic pullrequest configuraiton used to generate and/or apply Updatecli manifest
pullrequests:
  test:
    kind: github
    scmid: test
    #targets:
    #  - default
    #scmid: default # May conflicts with current code
    #targets:
    #  - goWorkflow # May conflicts with current code
    spec:
      labels:
        - chore
        - skip-changelog

# Provide autodiscovery configuration such as the scm or pullrequest config to use
autodiscovery:
  scmid: test
  pullrequestid: test
  crawlers:
    helm:
      #ignore:
      #  # - path: chart/epinio-ui/*
      #  # - path: ~/Projects/Epinio/helm-charts/chart/epinio/*
      #  # - path: chart/epinio/*
      #only:
      #  # - path: ~/Projects/Epinio/helm-charts/chart/epinio/*
      #  - path: chart/epinio/*

First plugin

As a first working scenario, I am using the Helm chart dependency that needs to be updated, but I am considering to use the same approach than for source/condition/target, where we could add more plugins.

For better visibility, I am planning to put this new plugin type in pkg/plugins/generics

So for a specific directory, I am looking for every Chart.yaml or Chart.yml that have dependencies listed. Then I return one manifest per dependency identified. This means that once merged, we would have a new way to specifies Updatecli manifest

1. Using yaml configuration
2. Generating them from a directory

Test

To test this pull request, you can run the following commands:

go build -o bin/updatecli . 
# Run from a directory with helm chart.
./bin/updatecli diff --config dumb.yaml

Additional Information

Tradeoff

Potential improvement

When we run updatecli diff or apply we set the default configuration to "updatecli.yaml" and we trigger an error if no configuration file is available. This is problematic with the builder because we are now in a situation where we may not need any configuration files.

Solution A: We don't trigger an error if "updatecli.yaml" is not available
Solution B: We require a new flag such as updatecli diff --auto

I would be more in favor of not triggering an error if updatecli.yaml is empty or doesn't exist

PS: Please note that the naming of package, struct, etc. is not definitive and evolve with my understanding of the current problem

another potential improvement is to run updatecli show to display detected manifest. Considering that updatecli show don't clone scm repositories, it won't work with remote repositories

[olblak]

For the same of this feature, I am hesitating between auto, generic, builder which explain why those three names are used in the pullrequest.

[olblak]

Please also note that this PR doesn't generate "pipelines" but "pipeline manifest"

[olblak]

The auto-discovery feature introduced in this pullrequest , means it should be possible to run updatecli commands without providing any manifest as updatecli diff could discovery available pipeline.
But due to some early design decision, by default if no manifests are provided then we fallback to "updatecli.yaml" and fail if it doesn't exist.

This commit, 11e468c refactor a bit so we first detect updatecli.yaml, to remain backward compatible, then updatecli.d if it's a directory, and finally we just try to autodiscover available pipelines

[olblak]

I think it would be better to split this pullrequest in two. One about the autodiscovery and a second one about helm chart dependency update

[olblak]

Here is the documentation that I started on the autodiscovery feature updatecli/website#428

[olblak]

I am just dropping this repository https://github.com/rancher/fleet-examples
To remind myself to use it in my test

[olblak]

CHANGELOG

The autodiscovery feature is a major feature that land the paving for a new way of using Updatecli

It introduces the following changes:

Fix Jsonschema Data

Due to a regression, the jsonschema expected Updatecli to start with a spec root key. This is fix now

Default config

By default, updatecli will look for pipeline manifest in updatecli.yaml or updatecli.d.
It doesn't return an error if no valid manifest could be found. This behavior change is important for the autodiscovery
as the autodiscovery can be used without providing manifest such as updatecli manifest show --local-autodiscovery

Manifest show

In version 0.27.0, we introduce sub-commands for manifest such as updatecli manifest upgrade
Now we are adding a new one updatecli manifest show which deprecates updatecli show
This subcommand is convenient to display and generate manifest based on the autodiscovery such as updatecli manifest show --local-autodiscovery

No manifest allowed

As mentioned in the previous section, Updatecli can analyze local files to generate and apply manifest.
This means that providing a valid manifest is no anymore mandatory for Updatecli
The following commands work

• updatecli diff --local-autodiscovery
• updatecli apply --local-autodiscovery
• updatecli show --local-autodiscovery
• updatecli manifest show --local-autodiscovery

Drop target in pullrequest spec
In the past, the pullrequest spec would rely on the field "targets" to know which target was bound to the pullrequest
This information could easily be generated from the context.

before

pullrequests:
  default:
    title: '[updatecli] Bump updatecli version to {{ source "latestVersion" }}'
    kind: github
    scmid: default
    targets:
      - bugReport
    spec:
      labels:
        - chore
        - skip-changelog

After

pullrequests:
  default:
    title: '[updatecli] Bump updatecli version to {{ source "latestVersion" }}'
    kind: github
    scmid: default
    spec:
      labels:
        - chore
        - skip-changelog

You can easily fix your manifest by running updatecli manifest upgrade --config ./updatecli.d to validate the changes and then
updatecli manifest upgrade --config <manifest name>.yaml --in-place

autodiscovery

WARNING: Autodiscovery can be enabled by using the flag --experimental. We decided to use the experimental to highlight that we are currently in a very early stage, and this feature can easily be modified based on user input.

The goal of this feature is to parse common files to try to generate and apply manifest. The goal is to reduce to need write and maintain manifest that could easily be generated by Updatecli based on context.
It can be configured from the same manifest by using the top-level key autodiscovery
More information is available on www.updatecli.io/docs/core/autodiscovery

Helm crawler

The first autodiscovery crawler named "helm", allows parsing common Helm chart configuration to update the following information.

• Helm dependencies defined in Chart.yaml
• Container images defined in values.yaml using the following format

image:
  repository: <image_name>
  tag: <tag>
images:
  <id>:
    repository: <image_name>
    tag: <image_tag>

title: Manifest title
scms
  # As before, it specifies scm configuration
pullrequests:
  # As before, it specifies pullrequest configuration
autodiscovery:
  # Optional
  # If one is specified, the updatecli analyzes the scm directory, instead of the local one
  scmid: <it references a scm configuration >

  # Optional
  pullrequest: <specifis a pullrequest id>

  # Override default crawlers configuration
  crawlers:
    helm:
      # Default true
      enabled: true

      # Optional
      ignore:
        - path: <specifies path to ignore>

      # Optional
      only:
        - path: <only explore this path>

      # OptionDefault scm directory or local if nothing specified
      rootdir: <override directory to analyze>

      # Auths specifies docker registry credentials
      auths:
          docker.io:
              username: <username>
              token: <token>
          ghcr.io:
              username: <username>
              token: <token>

Fleet Crawler

A second crawler has been added to help to put things in perspective, it updates helm version from fleet.yaml files

An example of autodiscovery manifest would be

title: Manifest title
scms
  # As before, it specifies scm configuration
pullrequests:
  # As before, it specifies pullrequest configuration
autodiscovery:
  # Optional
  # If one is specified, the updatecli analyzes the scm directory, instead of the local one
  scmid: <it references a scm configuration >

  # Optional
  pullrequest: <specifis a pullrequest id>

  # Override default crawlers configuration
  crawlers:
    rancher/fleet:
      # default true
      enabled: true

      # Optional
      ignore:
        - path: <specifies path to ignore>

      # Optional
      only:
        - path: <only explore this path>

      # OptionDefault scm directory or local if nothing specified
      rootdir: <override directory to analyze>