toYaml removes all quotes and yaml types breaks

[shal]

Output of helm version:

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.10-gke.0", GitCommit:"16ebd0de8e0ab2d1ef86d5b16ab1899b624a77cd", GitTreeState:"clean", BuildDate:"2018-03-20T20:21:01Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE

I got following problem:

File: values.yml

info:
  address: "0x1ddf249430c343f72f8b12848aa3f932538e3336"

File: configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
data:
  file.yml: |-
{{ toYaml .Values.info | indent 4 }}

After toYaml address will be not a string.

[bacongobbler]

@ashanaakh can you provide the output of helm template with this sample?

[shal]

@bacongobbler Sorry, I can't provide all output, because it is confidential information.

I see, that function changed type of 0x1ddf249430c343f72f8b19868ea4f932500e0000.

In values.yml it was in quotes, like

address: "0x1ddf249430c343f72f8b19868ea4f932500e0000"

After toYaml, I see

address: 0x1ddf249430c343f72f8b19868ea4f932500e0000

so configmap looks like this

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
data:
  file.yml: |-
    address: 0x1ddf249430c343f72f8b19868ea4f932500e0000

[bacongobbler]

oh I see what you mean. However, both are considered strings though according to the YAML spec though, right? That should still work as a string as single, double and unquoted values are accepted as strings according to the spec.

[shal]

@bacongobbler Unfortunately, 0x1ddf249430c343f72f8b19868ea4f932500e0000 is hexadecimal literal in YAML.

So it is kind of integer.

Please, take a look on YAML specification here.

Thanks!

[shal]

@bacongobbler

Here I see, that toYaml function just use dependency for marshaling.

https://github.com/kubernetes/helm/blob/aa98e7e3dd2356bce72e8e367e8c87e8085c692b/pkg/strvals/parser.go#L33-L40

I reproduced this bug using helm dependency github.com/ghodss/yaml.

package main

import (
	"fmt"
	"github.com/ghodss/yaml"
)

func main() {
	p := map[string]string{"key": "0x1ddf249430c343f72f8b19868ea4f932500e0000"}
	d, _ := yaml.Marshal(p)
	fmt.Print(string(d))
}

Output will be:

key: 0x1ddf249430c343f72f8b19868ea4f932500e0000

Actually this dependency has one more dependency, which is github.com/go-yaml/yaml, where this bug occurs.

[hmcmanus]

It might be a bit late now but I had the same problem but was able to get around it by copying what these guys did here:

https://github.com/kubernetes/charts/blob/master/incubator/elasticsearch-curator/values.yaml

[amir-hadi]

@shal You closed this issue, did the problem disappear for you? I see a similar behavior with toYaml which is caused problems for me right now.

[shal]

Hey @amir-hadi, I temporary resolved this issue by just copying all YAML files as config maps into the helm chart 😄

[jocko-wowza]

So I'm trying to us toYaml to dynamically inject custom Environment variables along side required environment variables in my container: definition in my Helm chart.

The problem is that regardless of whether the value is a numer or string, the requirement is that the environment value needs to be quoted. Because toYaml is unquoting it my helm chart is failing.

Anyone know how I can prevent this?

[jocko-wowza]

I solved it by not using toYaml ... That's a seemingly direct way to get data from a values file to append to my env: key, but this solution allows me to make sure all of the values are quoted:

container:
  env:
  - name: SOME_ENV_VARIABLE
     value: "foo"
  - name: SOME_OTHER_VARIABLE
     value: "bar"
{{- range .Values.some_value.env }}
  - name: {{ .name }}
    value: {{ .value | quote }}
{{- end }}

[rdxmb]

In my opinion this is a bug and not an question/support like it is actually labeled.

[rdxmb]

Can this issue be opened again?

[bacongobbler]

If someone wants to take a crack at this, here's where toYaml resides in the source code. Pull requests with unit tests welcome.

helm/pkg/engine/funcs.go

Lines 79 to 90 in bb33b92

	// toYAML takes an interface, marshals it to yaml, and returns a string. It will
	// always return a string, even on marshal error (empty string).
	//
	// This is designed to be called from a template.
	func toYAML(v interface{}) string {
	data, err := yaml.Marshal(v)
	if err != nil {
	// Swallow errors inside of a template.
	return ""
	}
	return strings.TrimSuffix(string(data), "\n")
	}

[bacongobbler]

But before I start I would like to understand which operation in helm is failing

It's likely the call to yaml.Marshal that is stripping quotes from strings on line 84. See above ^

The information provided in the OP should provide enough information to reproduce the issue as well as the expected output. I would start with writing a unit test for toYaml and see if you can reproduce the same issue before working on a fix.

Let me know if you need further clarification.

[bacongobbler]

helm template.

[bacongobbler]

You can also reproduce the bug in isolation, as described in an earlier comment.

[AdheipSingh]

i use this, works for me

{{ tpl $value $ | quote }}

[dfurtado]

Hi everyone, I would like to leave my contribution to this issue, I have investigated a little bit, and I don't think this is actually a bug.

The package github.com/ghodss/yaml (used by Helm) uses gopkg.in/yaml.v2 internally. The gopkg.in/yaml.v2 try to figure it out which type of data it is going to be parsed and written to the YAML output. This is done in the marshal function in encode.go, in case of a map it will iterate through the keys and values and call marshal function recursively for each key and value

The scenario described by this issue, we have a string value 0x1ddf249430c343f72f8b19868ea4f932500e0000, the marshal function will call a function called stringv which will decide how to output the value, for instance, if it should be quoted, unquoted and so on.

Inside stringv, the function resolve is called and it will try to parse the value using strconv.ParseInt and strconv.ParseUInt, however, the value 0x1ddf249430c343f72f8b19868ea4f932500e0000 cannot be parsed correctly and the resolver assume that the value can be written as plain text returning a tag:yaml.org,2002:str and the value returned will be unquoted.

If we use the same example to reproduce the bug but instead set the key with a valid value, it works:

package main

import (
	"fmt"
	"gopkg.in/yaml.v2"
)

func main() {
	p := map[string]string{"key": "0x1c8"}
	d, _ := yaml.Marshal(p)
	fmt.Print(string(d))
}

Output:

key: "0x1c8"

Trying to parse that value with the strconv.ParseInt you can see the error:

data, err := strconv.ParseInt("0x1ddf249430c343f72f8b19868ea4f932500e0000", 0, 64)
fmt.Println(data)
fmt.Println(err)

Result:

9223372036854775807
strconv.ParseInt: parsing "0x1ddf249430c343f72f8b19868ea4f932500e0000": value out of range

As you can see in the error message the value 9223372036854775807 is the max size of int64. That's why the parsed value is out of range.

So I don't think it is a bug since YAML package seems to be doing the right thing and the toYAML function in Helm is simply getting the results and returning it which is correct. It was just the value that was too long to be parsed to a int64 causing the value to be unquoted.

Maybe the post above using the | quote would be a option to force quoting the value even when the YAML parser thinks it should be in plain text.

Do you have any feedback on this findings? @bacongobbler

[franck102]

Just encountered this, it makes toYaml pretty useless for us. With this in values.yaml:

foo:
  bar: ":tricky-{}[]!#|>foobar%@"

this template

foo:
{{  .Values.foo | toYaml | indent 2 }}

loses the quotes:

foo:
  bar: :tricky-{}[]!#|>foobar%@

[bacongobbler]

see #8890.

[bacongobbler]

At least this should be done.

Please feel free to do so. Helm is a community project. The documentation can be found here.

[rdxmb]

Hm, I guess I do not know if I've understooden the necessary things to do that.
To be honest, I did not get the "way to do" it in this case.

Let's try:

$ cat values.yaml 
file: 
  address: "0x1ddf249430c343f72f8b12848aa3f932538e3336"
$ cat templates/dummy.yaml 
---
data:
  toYaml.yml: |
{{ toYaml .Values.file | indent 4 }}
$ helm template .
---
# Source: helm-stuff/templates/dummy.yaml
data:
  toYaml.yml: |
    address: 0x1ddf249430c343f72f8b12848aa3f932538e3336

$ cat values.yaml 
file: |
  address: "0x1ddf249430c343f72f8b12848aa3f932538e3336"
$ cat templates/dummy.yaml 
---
data:
  toYaml.yml: |
{{ tpl .Values.file $ | indent 4 }}
$ helm template .
---
# Source: helm-stuff/templates/dummy.yaml
data:
  toYaml.yml: |
    address: "0x1ddf249430c343f72f8b12848aa3f932538e3336"

Ok. This also works with

$ cat values.yaml 
env: |
  - name: address
    value: "0x1ddf249430c343f72f8b12848aa3f932538e3336"
$ cat templates/dummy.yaml 
---
env:
{{ tpl .Values.env $ }}
$ helm template .
---
# Source: helm-stuff/templates/dummy.yaml
env:
- name: address
  value: "0x1ddf249430c343f72f8b12848aa3f932538e3336"

Can we see tpl combining with | in the values as a solution for most of the use cases?

For this does not work:

$ cat values.yaml 
env: 
  - name: address
    value: "0x1ddf249430c343f72f8b12848aa3f932538e3336"
$ cat templates/dummy.yaml 
---
env:
{{ tpl .Values.env $ }}
$ helm template .
Error: template: helm-stuff/templates/dummy.yaml:3:14: executing "helm-stuff/templates/dummy.yaml" at <.Values.env>: wrong type for value; expected string; got []interface {}

Use --debug flag to render out invalid YAML

Sorry, I am kind of confused what the recommended way is to do things like that.

[FelipeEmerim]

Is there a solution to the library charts use case? Pretty much every template uses toYaml because of the merge function bellow:

{- define "library.util.merge" -}}
{{- $top := first . -}}
{{- $overrides := fromYaml (include (index . 1) $top) | default (dict ) -}}
{{- $tpl := fromYaml (include (index . 2) $top) | default (dict ) -}}
{{- toYaml (merge $overrides $tpl) -}}
{{- end -}}

Could this be rewritten in another way?

[SupornoChaudhury]

Can I also take a look at it? I went through the older comments and all. I can collaborate with you rdxmb.

[rdxmb]

please feel free @SupornoChaudhury

Sorry, I am kind of confused what the recommended way is to do things like that.

[SupornoChaudhury]

I spent a while going down a rabbit hole of marshal functions which led me to go reflect package which was being used for determining the type of var once the double quotes are removed.

this is the toYAML function:

func toYAML(v interface{}) string {
	data, err := yaml.Marshal(v)
	if err != nil {
		// Swallow errors inside of a template.
		return ""
	}
	return strings.TrimSuffix(string(data), "\n")
}

I am a little new to Go, but I was wondering if we can identify all content within the quotes using regexp and then before returning the output, we replace the content back within quotes.

[Input Text]
Hello:
  how: "are"
  you: "123"

[quoted text]
are, 123

the text identified as quoted at the start of the function will be padded with quotes again at the end of the toYAML function irrespective of their reflected type.

current toYAML output: hello=text,how=are,you=texter
desired toYAML output: hello="text",how=are,you="texter"

I believe that would not cause any previous implementation to break.
You can refer to this golang playground that I created for an example:

https://play.golang.org/p/Ms7pN-NxmvV

[almson]

I'm not sure if this should go under this issue, although it's definitely in the same genre.

YAML types (eg property: !MyClass {}) get stripped out, and actually replaced by empty objects. Very broken.

[djmnnit10]

I used single quotes around the double quotes to get double quotes in toYaml output.
SERVER_USE-FORWARD-HEADERS: '"true"'

[artemmikhalitsin]

Looks like the issue reported by OP happens because the number is too big to fit into uint64, so is treated as a string and hence rendered without quotes. I submitted a possible fix in go-yaml/yaml, but I'm not sure whether this will fix all the related issues.

Stripping quotes from strings is the expected behaviour of go-yaml/yaml, and should result in a valid yaml configuration

[theAkito]

In other words perhaps some documentation around how to use toYaml in a chart would be more sufficient.

I've actually searched for hours to find clear and unambiguous documentation about toYaml and I simply could not find it anywhere. Many people talk about using it in general and using it a certain way, but I could not find a single, actual and official definition of that feature. It seems like this is some kind of hack or hidden feature, nobody should be using.

A feature without documentation is almost like a feature that does not exist, because if nobody can be sure to use it precisely according to spec -- as there is no spec -- then the feature becomes useless, and unusable in scenarios, where you want to be absolutely sure, it does exactly what is expected.

This issue here, which has been open for such an enormously long time, pretty much wouldn't exist, if documentation would be proper.

I constantly break my teeth out when trying to find documentation on any of the Kubernetes tools, which are popular. Kubernetes has documentation, but considering how complicated that software is, it would need 100 times more documentation, than it has now. Traefik has terrible documentation. You can't know anything when reading their documentation.
With Helm it's actually the funniest. When browsing the documentation, it seems like there is a lot of information there, but if you really want to look something up and get to know a Helm feature, you can't get to know anything, as only a few features are touched and even then, it's not really explained, but only mentioned in a sentence, as if it were the most obvious thing in the world.

Basically, software without documentation becomes useless for most users. Too few people have the time or ability to browse through a thousand lines of Go source code, just to get the general gist of what is supposed to happen, regarding a certain functionality.

That said, I understand if it's not possible for the core maintainers to address every single issue in a timely manner. On the other hand, I think it's unfair to demand, that users, especially the ones who don't even know anything about how this works, are supposed to do work for this, like in this case, document this feature.

I would even document it, if I would get the point of it. However, I cannot get the point of it, as there is no information on it, beside the source code itself, offering that "feature".

A tiny step making this whole thing slightly less worse, is at least the following comment.

#4262 (comment)

At least, this gives a little insight into what is happening. Still, one would need to read the whole YAML implementation of this specific library for the Go language, to be really sure, what the spec of this feature is or at least, to know what should and should not happen.

In other words, changing the behavior of toYaml may NOT be desirable, and it may be more preferable to document its behavior so that we don't accidentally break charts that have worked around this behavior.

I also do not agree with this. First of all, where is the spec? If there is no spec, there is no definitive behaviour. I.e. it's already broken. Nobody can be sure, it does exactly what it is expected to do, as proven by this issue. As I said, this issue wouldn't exist, if proper documentation would be available.

To my knowledge, the current state of toYaml is, that its behaviour is undefined. So, there is no breaking it, as it is already broken.

[bacongobbler]

First of all, where is the spec?

https://yaml.org/spec/ is the canonical specification for YAML. If the yaml parser library does not conform to that specification, it is a bug.

toYaml should be pretty self-explanatory: it takes a value and converts it to YAML. Any issues related to how those values are parsed and passed back to the template engine would be a bug in the YAML parser itself, which is why I suggested people try reproducing the bug in isolation using the YAML parser library.

As mentioned before, the entirety of toYaml's code can be found here. Helm passes the value to the YAML parser, and returns the output.

Sounds like this is an upstream issue based on #4262 (comment). Thank you @artemmikhalitsin for contributing a fix upstream. If/when that fix is merged, we can update our dependencies to receive a fix.

Doesn't sound like there's any further action to take here on our end. Please follow up with the go-yaml/yaml folks on go-yaml/yaml#820. Thanks.

[aarontavio]

I used single quotes around the double quotes to get double quotes in toYaml output. SERVER_USE-FORWARD-HEADERS: '"true"'

This comment helped me achieve my goal. Thaaanks @djmnnit10!.
Maybe this helps somebody else with a similar problem like me that ends up here.

[ba32107]

I used single quotes around the double quotes to get double quotes in toYaml output. SERVER_USE-FORWARD-HEADERS: '"true"'

This doesn't work for me. The single quotes are just carried over to the final output :(

None of the proposed solutions work if you don't control the chart you are using. In my case it's:
https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/templates/extra-manifests.yaml

This allows adding any extra K8s object to the chart, but without quotes it's proving to be very challenging :/

---

EDIT: I finally found a workaround that works for my use-case (cronjob schedule). Instead of this:

schedule: '0 1 * * *' 

which results in:

schedule: 0 1 * * *

I can add whitespace between the quote and the expression:

schedule: ' 0 1 * * * ' 

and it will be rendered as:

schedule: ' 0 1 * * * ' 

Which K8s accepts, despite the extra whitespace (it does not accept a cron schedule without quotes). Hope this helps someone!

[mindw]

Not being able to keep the type of the original yaml is a very surprising behavior. One expects the string "1" to stay a string and not turn into a yaml integer 1:
The issue seems to be in "sigs.k8s.io/yaml" package in the Unmarshal function as the following program demonstrates:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
  e: "5"
`

func main() {

	// Unmarshal the YAML back into a Person struct.
	m := map[string]interface{}{}

	err = yaml.Unmarshal([]byte(data), &m)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(m)
}

/*
age: 30
name: John

map[a:Easy! b:map[c:2 d:[3 4] e:5]]
*/

The reference go-yaml v2 & v3 program (with the "e" addition) keeps the quotes just fine.

[andykcfung]

After trying for an hour to stop toYaml from removing the quotes that I wanted to keep, I found a solution for my specific usage:

To begin, I have this entry in my values.yaml (block 1)

global:
  properties:
     mad:
       person: 'My id: 1234'

where "1234" should actually come from another entry in my values.yaml (note that I hard coded the id "1234" above but this will be addressed by the solution later). The following entry is also in my values.yaml:

person:
  id: 1234

Now I needed to create a file named "mad.yaml" with the following content using toYaml (note the single code around the value) (target i):

mad:
  person: 'My id: 1234'

The code that I will use to create the file "mad.yaml" is like this (code A):

mad.yaml: |-
{{ tpl (toYaml .Values.global.properties) . | indent 4}}

The solution: I modified my original entry (i.e., block 1) to this:

global:
  properties:
     mad:
       person: '{{ printf "%s %d" "My id:" (.Values.person.id | int) }}'

and my code (code A) was able to produce what I wanted (i.e., target i).

Hope this helps. It took me much longer to write this up than to find the solution I needed.

[joejulian]

If your string has to have the single quotes, you could also use the squote filter

With a value of:

mystring: My id: 1234

this template:

{{ .Values.mystring | squote }}

produces:

'My id: 1234'

[RiccardoManzan]

I needed to put a yaml defined in a values file as a string with yaml format, and in order to do that i used toYaml.
Unfortunately in that yaml portion i had a string which contained some special characters like : and }
I solved by using the backslash \ before special characters.

In this way i was able to keep my escapes.

[almson]

Not being able to keep the type of the original yaml is a very surprising behavior. One expects the string "1" to stay a string and not turn into a yaml integer 1: The issue seems to be in "sigs.k8s.io/yaml" package in the Unmarshal function as the following program demonstrates:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
  e: "5"
`

func main() {

	// Unmarshal the YAML back into a Person struct.
	m := map[string]interface{}{}

	err = yaml.Unmarshal([]byte(data), &m)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(m)
}

/*
age: 30
name: John

map[a:Easy! b:map[c:2 d:[3 4] e:5]]
*/

The reference go-yaml v2 & v3 program (with the "e" addition) keeps the quotes just fine.

That's pretty garbage. Could also explain why people keep needing quotes even though the yaml spec doesn't require them. Eg. why does CronJob schedule need to be quoted? If Kubernetes won't fix its problems, then Helm should not insist "we're confirming to YAML, don't look at us!" Otherwise that's garbage behavior from both Helm and Kubernetes. What a crappy ecosystem.

The fix is pretty "simple." Stop using these parsers, and write (fork) your own that tracks the quote status of strings on Unmarshal and reproduces them on Marshal. Or is that too difficult for you?