Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YAML to JSON error fixed by cleaning local repository #3111

Closed
drewwells opened this issue Nov 7, 2017 · 14 comments
Closed

YAML to JSON error fixed by cleaning local repository #3111

drewwells opened this issue Nov 7, 2017 · 14 comments
Labels
bug Categorizes issue or PR as related to a bug.

Comments

@drewwells
Copy link

We have an issue that 1 in 10-20 charts will fail with YAML to JSON error. This chart does not change, it is just being packaged with a new name. The only known workaround is to wipe the local cache and helm init. This error appears to be independent of the current path being packaged. Removing all the templates still results in this YAML to JSON error occurring

-> % helm package tkm-chart
Successfully packaged chart and saved it to: /path/to/tkm-chart-1.2.11.12-5a43454f-dirty.tgz
Error: error converting YAML to JSON: yaml: line 214: could not find expected ':'
-> % mv ~/.helm ~/.helm.bak
-> % helm init --client-only
Creating /dir/.helm
Creating /dir/.helm/repository
Creating /dir/.helm/repository/cache
Creating /dir/.helm/repository/local
Creating /dir/.helm/plugins
Creating /dir/.helm/starters
Creating /dir/.helm/cache/archive
Creating /dir/.helm/repository/repositories.yaml
$HELM_HOME has been configured at /dir/.helm.
Not installing tiller due to 'client-only' flag having been set
Happy Helming!

-> % helm package tkm-chart
Successfully packaged chart and saved it to: /path/to/tkm-chart-1.2.11.12-5a43454f-dirty.tgz
-> % rm -rf ~/.helm
-> % mv ~/.helm.bak ~/.helm
-> % helm package tkm-chart
Successfully packaged chart and saved it to: /path/to/tkm-chart-1.2.11.12-5a43454f-dirty.tgz
Error: error converting YAML to JSON: yaml: line 214: could not find expected ':'
@jascott1
Copy link
Contributor

jascott1 commented Nov 7, 2017

Hi @drewwells

What versions of Helm and k8s are you running? Also does this occur for any chart after n number of installs or just this chart?

@drewwells
Copy link
Author

-> % helm version
Client: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}

-> % k version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T21:07:53Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

foolishly, I deleted my broken ~/.helm. I will get back to you on any chart failing or just this one.

@drewwells
Copy link
Author

This happens for any chart that you attempt to package after the local repo is corrupted

-> % helm --home ~/.helm.bad package secured-prometheus -d build/
Successfully packaged chart and saved it to: build/secured-prometheus-0.2.2.tgz
Error: error converting YAML to JSON: yaml: line 702: could not find expected ':'

Same line number on every error message. I've kept the broken repo this time, for further debugging (but can't share it)

@wknapik
Copy link

wknapik commented Nov 30, 2017

I've been seeing this error from time to time as well. It's really annoying when it happens in CI jobs. Never thought to delete the .helm directory before packaging. Could be a workaround...

% head -n1 /etc/os-release
NAME="Arch Linux"
% helm version
Client: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
% kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.8", GitCommit:"e8c167a115ec662726904265d17f75a6d79d78d8", GitTreeState:"clean", BuildDate:"2017-10-01T00:01:59Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
% 

@wknapik
Copy link

wknapik commented Dec 8, 2017

At this point I run rm -fr ~/.helm in CI pipelines, as a workaround, but it still happens occasionally.

@cory-klein
Copy link

cory-klein commented Jan 24, 2018

We just saw this exact same issue today and it was indeed resolved by removing ~/.helm. However as a long-term solution we've added this to our Jenkinsfile:

environment {
  HELM_HOME = '${env.WORKSPACE}/.helm'
}

// Or the below for scripted pipelines

env.HELM_HOME = "${env.WORKSPACE}/.helm"

Our build is configured to reuse some contents of the workspace, so we also added this sanity check:

sh """
if [ -d ${env.HELM_HOME} ]; then
  echo 'Helm directory from previous build detected. Removing now.'
  rm -rf ${env.HELM_HOME}
else
  echo 'Fresh helm directory detected.'
fi
"""

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@wknapik
Copy link

wknapik commented Apr 25, 2018

/remove-lifecycle stale

@wknapik
Copy link

wknapik commented Nov 1, 2018

Still a problem as of helm v2.11.0.

@bacongobbler bacongobbler added bug Categorizes issue or PR as related to a bug. and removed question/support labels Nov 1, 2018
@aubm
Copy link

aubm commented Apr 3, 2019

I managed to reproduce the bug.

It appears that too many concurrent helm package operations will eventually leave the local index.yaml file corrupted.

Here is the sample go code that I use to reproduce:

package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"os/exec"
	"sync"
)

var (
	nbConcurrentProcesses int
)

func init() {
	flag.IntVar(&nbConcurrentProcesses, "n", 10, "nb concurrent processes")
	flag.Parse()
}

func main() {
	wg := sync.WaitGroup{}
	for i:= 0; i < nbConcurrentProcesses; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			runHelmPackage(i)
		}(i)
	}
	wg.Wait()
}

func runHelmPackage(i int) {
	output, err := os.Create(fmt.Sprintf("output_process_%v.txt", i))
	exitOnError(err)
	defer output.Close()

	// helm package /tmp/cerebro
	cmd := exec.Command("helm", "package", "/tmp/cerebro")
	cmd.Stdout = output
	cmd.Stderr = output
	if err := cmd.Run(); err != nil {
		fmt.Printf("failed to run helm install: %v\n", err)
		return
	}

	fmt.Println("it's all good baby")
}

func exitOnError(err error) {
	if err == nil {
		return
	}
	log.Fatal(err.Error())
}
# download this directory to /tmp/cerebro: https://github.com/helm/charts/tree/master/stable/cerebro
go run main.go -n 500

@wknapik
Copy link

wknapik commented Apr 3, 2019

For me this happens with some regularity when running 17 concurrent helm package invocations. I'm also working around #2258 by retrying failed calls up to 2 times (for a total of 3), so the actual number might be higher.

This bug was last triggered in my CI today, with helm version 2.12.3 on Linux.

@evisong
Copy link

evisong commented Apr 6, 2019

In my case, we're triggering 8 helm package commands in parallel on an AWS EC2 instance.

According to the debug log:

[debug] Setting version to 1.0.0
[debug] Setting appVersion to 1.0.0
Successfully packaged chart and saved it to: target/one_chart-1.0.0.tgz
[debug] Successfully saved target/one_chart-1.0.0.tgz to /root/.helm/repository/local

[debug] Setting version to 1.0.0
[debug] Setting appVersion to 1.0.0
Successfully packaged chart and saved it to: target/the_other_chart.tgz
Error: error converting YAML to JSON: yaml: line 14: could not find expected ':'

All helm package commands are able to create the helm tgz. However, the successful ones manage to save the *.tgz to local helm repo, but the failure ones throw error at this step.

We have other pipeline steps to upload helm charts to a centralized repo, so I'm adding --save=false flag to bypass the local repo.

This workaround is running for days, so far so good.

@wknapik
Copy link

wknapik commented Apr 8, 2019

Thanks for the hint @evisong.

@bacongobbler
Copy link
Member

I haven't seen any movement from the community to try and fix this issue, so I'm closing this as inactive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

9 participants