Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: allow helm overrides from valuesfile #594

Merged
merged 10 commits into from
May 21, 2024
29 changes: 27 additions & 2 deletions docs/overrides.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ packages:
overrides:
helm-overrides-component:
podinfo:
valuesFiles:
- file: values.yaml
values:
- path: "replicaCount"
value: 2
Expand All @@ -69,7 +71,13 @@ packages:
default: "purple"
```

This bundle will deploy the `helm-overrides-package` Zarf package and override the `replicaCount` and `ui.color` values in the `podinfo` chart. The `values` can't be modified after the bundle has been created. However, at deploy time, users can override the `UI_COLOR` and other `variables` using a environment variable called `UDS_UI_COLOR` or by specifying it in a `uds-config.yaml` like so:
```yaml
#values.yaml
podAnnotations:
customAnnotation: "customValue"
```

This bundle will deploy the `helm-overrides-package` Zarf package and override the `replicaCount`, `ui.color`, and `podAnnotations` values in the `podinfo` chart. The `values` can't be modified after the bundle has been created. However, at deploy time, users can override the `UI_COLOR` and other `variables` using a environment variable called `UDS_UI_COLOR` or by specifying it in a `uds-config.yaml` like so:

```yaml
variables:
Expand All @@ -92,6 +100,8 @@ packages:
overrides:
helm-overrides-component: # component name inside of the helm-overrides-package Zarf pkg
podinfo: # chart name from the helm-overrides-component component
valuesFiles:
- file: values.yaml
values:
- path: "replicaCount"
value: 2
Expand All @@ -102,7 +112,16 @@ packages:
default: "purple"
```

In this example, the `helm-overrides-package` Zarf package has a component called `helm-overrides-component` which contains a Helm chart called `podinfo`; note how these names are keys in the `overrides` block. The `podinfo` chart has a `replicaCount` value that is overridden to `2` and a variable called `UI_COLOR` that is overridden to `purple`.
```yaml
#values.yaml
podAnnotations:
customAnnotation: "customValue"
```
In this example, the `helm-overrides-package` Zarf package has a component called `helm-overrides-component` which contains a Helm chart called `podinfo`; note how these names are keys in the `overrides` block. The `podinfo` chart has a `replicaCount` value that is overridden to `2`, a `podAnnotations` value that is overridden to include `customAnnotation: "customValue"` and a variable called `UI_COLOR` that is overridden to `purple`.

### Values Files

The `valuesFiles` in an `overrides` block are a list of `file`'s. It allows users to override multiple values in a Zarf package component's underlying Helm chart, by providing a file with those values instead of having to include them all indiviually in the `overrides` block.

### Values

Expand Down Expand Up @@ -160,6 +179,12 @@ packages:
value: ${COLOR}
```

#### Value Precedence
Value precedence is as follows:
1. The `values` in an `overrides` block
1. `values` set in the last `valuesFile` (if more than one specified)
1. `values` set in the previous `valuesFile` (if more than one specified)

### Variables
Variables are similar to [values](#values) in that they allow users to override values in a Zarf package component's underlying Helm chart; they also share a similar syntax. However, unlike `values`, `variables` can be overridden at deploy time. For example, consider the `variables` key in the following `uds-bundle.yaml`:

Expand Down
73 changes: 73 additions & 0 deletions src/pkg/bundle/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ import (
"github.com/AlecAivazis/survey/v2"
"github.com/defenseunicorns/uds-cli/src/config"
"github.com/defenseunicorns/uds-cli/src/pkg/bundler"
"github.com/defenseunicorns/uds-cli/src/types"
zarfConfig "github.com/defenseunicorns/zarf/src/config"
"github.com/defenseunicorns/zarf/src/pkg/interactive"
"github.com/defenseunicorns/zarf/src/pkg/message"
"github.com/defenseunicorns/zarf/src/pkg/utils"
"github.com/pterm/pterm"
"helm.sh/helm/v3/pkg/chartutil"
)

// Create creates a bundle
Expand All @@ -26,6 +28,11 @@ func (b *Bundle) Create() error {
return err
}

// Populate values from valuesFiles if provided
decleaver marked this conversation as resolved.
Show resolved Hide resolved
if err := b.processValuesFiles(); err != nil {
return err
}

// confirm creation
if ok := b.confirmBundleCreation(); !ok {
return fmt.Errorf("bundle creation cancelled")
Expand Down Expand Up @@ -107,3 +114,69 @@ func (b *Bundle) confirmBundleCreation() (confirm bool) {
}
return true
}

// processValuesFiles reads values from valuesFiles and updates the bundle with the override values
func (b *Bundle) processValuesFiles() error {
// Populate values from valuesFiles if provided
for i, pkg := range b.bundle.Packages {
for componentName, overrides := range pkg.Overrides {
for chartName, bundleChartOverrides := range overrides {
valuesFilesToMerge := make([][]types.BundleChartValue, 0)
// Iterate over valuesFiles in reverse order to ensure subsequent value files takes precedence over previous ones
for _, valuesFile := range bundleChartOverrides.ValuesFiles {
// Check relative vs absolute path
fileName := filepath.Join(b.cfg.CreateOpts.SourceDirectory, valuesFile)
if filepath.IsAbs(valuesFile) {
decleaver marked this conversation as resolved.
Show resolved Hide resolved
fileName = valuesFile
}
// read values from valuesFile
values, err := chartutil.ReadValuesFile(fileName)
if err != nil {
return err
}
if len(values) > 0 {
// populate BundleChartValue slice to use for merging existing values
valuesFileValues := make([]types.BundleChartValue, 0, len(values))
for key, value := range values {
valuesFileValues = append(valuesFileValues, types.BundleChartValue{Path: key, Value: value})
}
valuesFilesToMerge = append(valuesFilesToMerge, valuesFileValues)
}
}
override := b.bundle.Packages[i].Overrides[componentName][chartName]
// add override values to the end of the list of values to merge since we want them to take precedence
valuesFilesToMerge = append(valuesFilesToMerge, override.Values)
override.Values = mergeBundleChartValues(valuesFilesToMerge...)
b.bundle.Packages[i].Overrides[componentName][chartName] = override
}
}
}
return nil
}

// mergeBundleChartValues merges lists of BundleChartValue using the values from the last list if there are any duplicates
// such that values from the last list will take precedence over the values from previous lists
func mergeBundleChartValues(bundleChartValueLists ...[]types.BundleChartValue) []types.BundleChartValue {
merged := make([]types.BundleChartValue, 0)
paths := make(map[string]bool)

// Iterate over each list in order
for _, bundleChartValues := range bundleChartValueLists {
// Add entries from the current list to the merged list, overwriting any existing entries
for _, bundleChartValue := range bundleChartValues {
if _, ok := paths[bundleChartValue.Path]; ok {
// Remove the existing entry from the merged list
for i, mergedValue := range merged {
if mergedValue.Path == bundleChartValue.Path {
merged = append(merged[:i], merged[i+1:]...)
decleaver marked this conversation as resolved.
Show resolved Hide resolved
break
}
}
}
merged = append(merged, bundleChartValue)
paths[bundleChartValue.Path] = true
}
}

return merged
}
44 changes: 44 additions & 0 deletions src/test/bundles/07-helm-overrides/values-file/uds-bundle.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
kind: UDSBundle
metadata:
name: helm-values-file
description: testing a bundle with Helm overrides
version: 0.0.1

packages:
- name: helm-overrides
path: "../../../packages/helm"
ref: 0.0.1

overrides:
podinfo-component:
unicorn-podinfo:
valuesFiles:
- values.yaml
decleaver marked this conversation as resolved.
Show resolved Hide resolved
- values2.yaml
values:
- path: "podinfo.replicaCount"
value: 2
variables:
- name: log_level
path: "podinfo.logLevel"
description: "Set the log level for podinfo"
default: "debug" # not overwritten!
- name: ui_color
path: "podinfo.ui.color"
description: "Set the color for podinfo's UI"
default: "blue"
- name: UI_MSG
path: "podinfo.ui.message"
description: "Set the message for podinfo's UI"
- name: SECRET_VAL
path: "testSecret"
description: "testing a secret value"
- name: SECURITY_CTX
path: "podinfo.securityContext"
description: "testing an object"
default:
runAsUser: 1000
runAsGroup: 3000
- name: HOSTS
path: "podinfo.ingress.hosts"
description: "just testing a a list of objects (doesn't actually do ingress things)"
12 changes: 12 additions & 0 deletions src/test/bundles/07-helm-overrides/values-file/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
podinfo.replicaCount: 3
podinfo.tolerations:
- key: "unicorn"
operator: "Equal"
value: "defense"
effect: "NoSchedule"
- key: "uds"
operator: "Equal"
value: "true"
effect: "NoSchedule"
podinfo.podAnnotations:
customAnnotation: "customValue"
3 changes: 3 additions & 0 deletions src/test/bundles/07-helm-overrides/values-file/values2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
podinfo.replicaCount: 4
podinfo.podAnnotations:
customAnnotation: "customValue2"
39 changes: 39 additions & 0 deletions src/test/e2e/variable_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,45 @@ func TestBundleWithHelmOverrides(t *testing.T) {
remove(t, bundlePath)
}

func TestBundleWithHelmOverridesValuesFile(t *testing.T) {
deployZarfInit(t)
e2e.HelmDepUpdate(t, "src/test/packages/helm/unicorn-podinfo")
e2e.CreateZarfPkg(t, "src/test/packages/helm", false)
bundleDir := "src/test/bundles/07-helm-overrides/values-file"
bundlePath := filepath.Join(bundleDir, fmt.Sprintf("uds-bundle-helm-values-file-%s-0.0.1.tar.zst", e2e.Arch))
err := os.Setenv("UDS_CONFIG", filepath.Join("src/test/bundles/07-helm-overrides", "uds-config.yaml"))
require.NoError(t, err)

createLocal(t, bundleDir, e2e.Arch)
deploy(t, bundlePath)

// test values overrides
t.Run("check values overrides", func(t *testing.T) {
cmd := strings.Split("zarf tools kubectl get deploy -n podinfo unicorn-podinfo -o=jsonpath='{.spec.replicas}'", " ")
outputNumReplicas, _, err := e2e.UDS(cmd...)
require.Equal(t, "'2'", outputNumReplicas)
require.NoError(t, err)
})

t.Run("check object-type override in values", func(t *testing.T) {
cmd := strings.Split("zarf tools kubectl get deployment -n podinfo unicorn-podinfo -o=jsonpath='{.spec.template.metadata.annotations}'", " ")
annotations, _, err := e2e.UDS(cmd...)
require.Contains(t, annotations, "\"customAnnotation\":\"customValue2\"")
require.NoError(t, err)
})

t.Run("check list-type override in values", func(t *testing.T) {
cmd := strings.Split("zarf tools kubectl get deployment -n podinfo unicorn-podinfo -o=jsonpath='{.spec.template.spec.tolerations}'", " ")
tolerations, _, err := e2e.UDS(cmd...)
require.Contains(t, tolerations, "\"key\":\"uds\"")
require.Contains(t, tolerations, "\"value\":\"defense\"")
require.Contains(t, tolerations, "\"key\":\"unicorn\"")
require.Contains(t, tolerations, "\"effect\":\"NoSchedule\"")
require.NoError(t, err)

})
}

func TestBundleWithDupPkgs(t *testing.T) {
deployZarfInit(t)
e2e.SetupDockerRegistry(t, 888)
Expand Down
18 changes: 4 additions & 14 deletions src/types/bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,28 +28,18 @@

// BundleChartOverrides represents a Helm chart override to set via UDS variables
type BundleChartOverrides struct {
Values []BundleChartValue `json:"values,omitempty" jsonschema:"description=List of Helm chart values to set statically"`
Variables []BundleChartVariable `json:"variables,omitempty" jsonschema:"description=List of Helm chart variables to set via UDS variables"`
Namespace string `json:"namespace,omitempty" jsonschema:"description=The namespace to deploy the Helm chart to"`

// EXPERIMENTAL, not yet implemented
//ValueFiles []BundleChartValueFile `json:"value-files,omitempty" jsonschema:"description=List of Helm chart value files to set statically"`
Values []BundleChartValue `json:"values,omitempty" jsonschema:"description=List of Helm chart values to set statically"`
Variables []BundleChartVariable `json:"variables,omitempty" jsonschema:"description=List of Helm chart variables to set via UDS variables"`
Namespace string `json:"namespace,omitempty" jsonschema:"description=The namespace to deploy the Helm chart to"`
ValuesFiles []string `json:"valuesFiles,omitempty" jsonschema:"description=List of Helm chart value file paths to set statically"`
}

// BundleChartValue represents a Helm chart value to path mapping to set via UDS variables
type BundleChartValue struct {

Check warning on line 37 in src/types/bundle.go

View workflow job for this annotation

GitHub Actions / validate

exported type BundleChartValue should have comment or be unexported
Path string `json:"path" jsonschema:"name=Path to the Helm chart value to set. The format is <chart-value>, example=controller.service.type"`
Value interface{} `json:"value" jsonschema:"name=The value to set"`
}

// BundleChartValueFile - EXPERIMENTAL - represents a Helm chart value file to override
type BundleChartValueFile struct {
Path string `json:"path" jsonschema:"name=Path to the Helm chart to set. The format is <component>/<chart-name>, example=my-component/my-cool-chart"`
File string `json:"file" jsonschema:"name=The path to the values file to add to the Helm chart"`
}

// BundleChartVariable - EXPERIMENTAL - represents a Helm chart variable and its path
type BundleChartVariable struct {

Check warning on line 42 in src/types/bundle.go

View workflow job for this annotation

GitHub Actions / validate

exported type BundleChartVariable should have comment or be unexported
Path string `json:"path" jsonschema:"name=Path to the Helm chart value to set. The format is <chart-value>, example=controller.service.type"`
Name string `json:"name" jsonschema:"name=Name of the variable to set"`
Description string `json:"description,omitempty" jsonschema:"name=Description of the variable"`
Expand Down
7 changes: 7 additions & 0 deletions uds.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,13 @@
"namespace": {
"type": "string",
"description": "The namespace to deploy the Helm chart to"
},
"valuesFiles": {
"items": {
"type": "string"
},
"type": "array",
"description": "List of Helm chart value file paths to set statically"
}
},
"additionalProperties": false,
Expand Down
35 changes: 35 additions & 0 deletions zarf.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,41 @@
},
"type": "array",
"description": "List of local values file paths or remote URLs to include in the package; these will be merged together when deployed"
},
"variables": {
"items": {
"$schema": "http://json-schema.org/draft-04/schema#",
"$ref": "#/definitions/ZarfChartVariable"
},
"type": "array",
"description": "[alpha] List of variables to set in the Helm chart"
}
},
"additionalProperties": false,
"type": "object",
"patternProperties": {
"^x-": {}
}
},
"ZarfChartVariable": {
"required": [
"name",
"description",
"path"
],
"properties": {
"name": {
"pattern": "^[A-Z0-9_]+$",
"type": "string",
"description": "The name of the variable"
},
"description": {
"type": "string",
"description": "A brief description of what the variable controls"
},
"path": {
"type": "string",
"description": "The path within the Helm chart values where this variable applies"
}
},
"additionalProperties": false,
Expand Down
Loading