feat: state values

[mumoshu]

This adds values to state files as proposed in #640.

values:
- key1: val1
- defaults.yaml

environments:
  default:
  - values:
    - environments/default.yaml
  production:
  - values:
    - environments/production.yaml

{{ .Valuese.key1 }} evaluates to val1 if and only if it is not overrode via the production or the default env, or command-line args.

Resolves #640

[mumoshu]

@sgandon Unit tests are passing, but I'm not confident if it works.

Would you mind trying?

[mumoshu]

This is worth being followed-up with documentation updates once confirmed to work.

[sgandon]

@mumoshu I'll give it a try.

[sgandon]

From our initial tests this is working fine.

[mumoshu]

@sgandon Great! Thanks for reporting

[dudicoco]

Hi, this is a great feature that should be added to the documentation.
In addition, it seems that the exec function is not working within values.
The following is working when put into environment values but not when put into values:

files: {{ exec "sh" (list "-c" "find .. -name release.yaml | jq -R .| jq -s .") }}

[mumoshu]

@dudicoco Hey! Thanks for the feedback. Doc issue submitted: #1043

Re exec, what's the exact error you're seeing? It should work as long as you aren't doing fancy things with double-rendering.

[dudicoco]

@mumoshu the error is:
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:53:8: executing "stringTemplate" at <$.Values.test>: map has no entry for key "test"

Getting this even when performing a simple exec function:

values:
- test: {{ exec "sh" (list "-c" "echo test") }}

[mumoshu]

@dudicoco Where's your values section located? Please show me a fuller example so that I can see where things went wrong :)

[mumoshu]

Perhaps you're missing ---?

cat <<EOF > helmfile.yaml
values:
- test: {{ exec "sh" (list "-c" "echo test") }}
---
releases:
- name: myapp-{{ .Values.test }}
EOF

$ helmfile build

#  Source: helmfile.yaml

filepath: helmfile.yaml
values:
- test: test
releases:
- name: myapp-test
templates: {}

[dudicoco]

Adding --- resolved the issue, thanks!
Can you please explain the rendering behaviour?

[mumoshu]

Nice! Glad to hear it worked 😄

Regarding the behavior, this doesn't work:

values:
- test: {{ exec "sh" (list "-c" "echo test") }}
releases:
- name: myapp-{{ .Values.test }}

It's due to a chicken-and-egg problem. To render {{ .Values.test }}, Helmfile needs the value of .Values.test loaded. But to load .Values.test helmfile needs to parse helmfile.yaml as YAML. To parse it as YAML, it firstly renders it as a go template, but to render it you need the value of .Values.test...

[mumoshu]

#932 is the long term solution to avoid this (super unintuitive) behavior. I'd appreciate any feedback on the issue so that I am more confident with the solution and start implementing it.

[dudicoco]

Thanks for the info @mumoshu, i'll definitely checkout the issue when I get a chance.
As a side note, I noticed that when running helmfile build the helm repo password gets rendered (I've added it as requiredEnv). Is this intentional? I would assume it's best not to include sensitive information in the helmfile build output.

[mumoshu]

Yep it's intentional as of today. In near future I'd like to extend #906 to helmfile.yaml and state values, so that helmfile build output only contains references to secrets.

[mumoshu]

Thank you for the feedback though! It helps prioritizing what I should do next.

[dudicoco]

Thanks @mumoshu.
If I could pick your brain a bit more, I've tried converting the following:

files: {{ exec "sh" (list "-c" "find .. -name release.yaml | jq -R .| jq -s .") }}

To:

files: {{ exec "sh" (list "-c" "find .. -name release.yaml) | splitList "\n" }}

However, I end up with a list with no commas which I can't iterate over with the range function.
Any ideas how to handle this?
Thanks

[mumoshu]

I guess you would want this:

files:
{{ $f := range exec "sh" (list "-c" "find .. -name release.yaml) | splitList "\n" }}
- {{ $f }}
{{ end }}

[dudicoco]

Unfortunately it doesn't work:
"error during helmfile.yaml.part.0 parsing: template: stringTemplate:33: unexpected in command"

[mumoshu]

Sorry but that error doesn't make sense to me without context.

Are you by any chance just missing closing doublequote after release.yaml?

[mumoshu]

files:
{{ $i, $f := range exec "sh" (list "-c" "find .. -name release.yaml") | splitList "\n" }}
- {{ $f }}
{{ end }}

[dudicoco]

@mumoshu couldn't get it to work with the above, do you need any other information?

[mumoshu]

@dudicoco Hey! Would u mind sharing the output from helmfile --log-level=debug build in GitHub Gist?

[dudicoco]

processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{stagin~g map[] map[]}, overrode=<nil>
first-pass uses: &{stagin~g map[] map[]}
first-pass rendering input of "helmfile.yaml.part.0":
 0:
 1: helmDefaults:
 2:   # Additional and global args passed to helm.
 3:   args: []
 4:     # - "--set k=v"
 5:   # Wait for k8s resources.
 6:   wait: true
 7:   # Time in seconds to wait for any individual Kubernetes operation (like Jobs for hooks, and waits on pod/pvc/svc/deployment readiness).
 8:   timeout: 300
 9:   # Performs pods restart for the resource if applicable.
10:   recreatePods: false
11:   # Forces resource update through delete/recreate if patch fails.
12:   force: false
13:   # Restores previous state in case of failed release.
14:   atomic: false
15:   # Cleans up any new resources created during a failed release.
16:   cleanupOnFail: false
17:
18: repositories:
19: - name: stable
20:   url: https://kubernetes-charts.storage.googleapis.com
21: - name: incubator
22:   url: https://kubernetes-charts-incubator.storage.googleapis.com
23: - name: ****
24:   url: https://raw.githubusercontent.com/****
25:   username: {{ requiredEnv "GITHUB_USER" }}
26:   password: {{ requiredEnv "GITHUB_TOKEN"}}
27:
28: values:
29: {{ $command := printf "find . -name release-%s.yaml | jq -R .| jq -s ." .Environment.Name }}
30: - files: {{ exec "sh" (list "-c" $command) }}
31: - files2:
32: {{ $i, $f := range exec "sh" (list "-c" "find .. -name release.yaml") | splitList "\n" }}
33: - {{ $f }}
34: {{ end }}
35:
36: environments:
37:   {{ .Environment.Name }}:
38:     values:
39:     - ../global/global-values-{{ .Environment.Name }}.yaml
40:

template syntax error: template: stringTemplate:33: too many declarations in command
first-pass produced: &{stagin~g map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {stagin~g map[] map[]}
second-pass rendering failed, input of "helmfile.yaml.part.0":
 0:
 1: helmDefaults:
 2:   # Additional and global args passed to helm.
 3:   args: []
 4:     # - "--set k=v"
 5:   # Wait for k8s resources.
 6:   wait: true
 7:   # Time in seconds to wait for any individual Kubernetes operation (like Jobs for hooks, and waits on pod/pvc/svc/deployment readiness).
 8:   timeout: 300
 9:   # Performs pods restart for the resource if applicable.
10:   recreatePods: false
11:   # Forces resource update through delete/recreate if patch fails.
12:   force: false
13:   # Restores previous state in case of failed release.
14:   atomic: false
15:   # Cleans up any new resources created during a failed release.
16:   cleanupOnFail: false
17:
18: repositories:
19: - name: stable
20:   url: https://kubernetes-charts.storage.googleapis.com
21: - name: incubator
22:   url: https://kubernetes-charts-incubator.storage.googleapis.com
23: - name: ****
24:   url: https://raw.githubusercontent.com/****
25:   username: {{ requiredEnv "GITHUB_USER" }}
26:   password: {{ requiredEnv "GITHUB_TOKEN"}}
27:
28: values:
29: {{ $command := printf "find . -name release-%s.yaml | jq -R .| jq -s ." .Environment.Name }}
30: - files: {{ exec "sh" (list "-c" $command) }}
31: - files2:
32: {{ $i, $f := range exec "sh" (list "-c" "find . -name release.yaml") | splitList "\n" }}
33: - {{ $f }}
34: {{ end }}
35:
36: environments:
37:   {{ .Environment.Name }}:
38:     values:
39:     - ../global/global-values-{{ .Environment.Name }}.yaml
40:

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:33: too many declarations in command
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:33: too many declarations in command

[mumoshu]

Ah sry it should be range $i, $f := rather than $i, $f := range

[dudicoco]

Thanks @mumoshu, it worked!
This is what I ended up with:

values:
{{ $command := printf "find . -name release-%s.yaml" .Environment.Name }}
- files:
  {{ range $i, $f := exec "sh" (list "-c" $command) | splitList " " }}
  - {{ $f }}
  {{ end }}

I think it's an amazing way to both load up all of your helmfiles and template them within your main helmfile.
This is due to the fact that the helmfiles: feature renders each file independently and doesn't allow you to define dependencies (needs) between releases nor does it allow you to template the helmfiles with native go templating.

[dudicoco]

Ok I spoke too soon :)
Seems that it doesn't work and I run into the exact same issue - I end up with a single list item:

values:
- files:
  - ./devops/php-myadmin/release.yaml
    ./devops/kibana/release.yaml
    ./devops/filebeat/release.yaml
    ./devops/cerebro/release.yaml

[dudicoco]

So the issue seems to be with splitList " ", if I use splitList "\n" I get a proper list but with an empty item at the end which causes an error.

[dudicoco]

Solved with the following:

- files:
  {{ range $i, $f := exec "sh" (list "-c" $command) | splitList "\n" }}
  {{ if $f }}
  - {{ $f }}
  {{ end }}
  {{ end }}