Failing on : jx step helm release

[hoaitang]

Summary

We are having the jx cluster running fine, suddently, the "jx step helm release" start failing with error below:

error: failed to build dependencies for chart from directory '.': failed to add chart repositories:
failed to add the Helm repository with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io':
failed to add the repository 'storage.googleapis.com' with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io':
failed to run 'helm repo add storage.googleapis.com https://storage.googleapis.com/chartmuseum.jenkins-x.io'
command in directory '.', output: 'Error: Couldn't load repositories file (/tekton/home/.helm/repository/repositories.yaml).

If you have any inside , please advice,
Much appreciate

[PuckOut]

same here

[vilbert]

same here, any solution?

[pod2metra]

Same story.

[aure-olli]

Got the same problem.

The helm repo https://storage.googleapis.com/chartmuseum.jenkins-x.io is now dead, should now use https://charts.helm.sh/stable
This is the case in the lastest helm, but the helm used in Jenkins-X docker images is highly outdated.

docker run gcr.io/jenkinsxio/builder-go:2.1.150-769 helm version
> Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}

docker run gcr.io/jenkinsxio/builder-go:2.1.150-769 helm init --help | grep 'stable-repo-url'
>       --stable-repo-url string            URL for stable repository (default "https://kubernetes-charts.storage.googleapis.com")

There are 2 ways to fix that:

• the most obvious one is to update the helm2 version used in all the images
• can also set jx to force the repo https://charts.helm.sh/stable with the option --stable-repo-url https://charts.helm.sh/stable in HelmCli.Init: https://github.com/jenkins-x/jx/blob/master/pkg/helm/helm_cli.go#L148

args = append(args, "--stable-repo-url", "https://charts.helm.sh/stable")

In the meanwhile, I'll create my own docker images to make it work.

[aure-olli]

OK apparently the helm version of the image doesn't matter because it's downloaded again.
The version is hardcoded here: https://github.com/jenkins-x/jx/blob/master/pkg/packages/packages.go#L32
And yes, it means that helm made a critical change without backward compatibility in only one change ...

[pod2metra]

@aure-olli can you, please, post a code snippet about new docker image building or share more info about fix?

[aure-olli]

OK got it to work on my cluster

Build a new jx

git clone https://github.com/jenkins-x/jx
cd jx
nano +32 pkg/packages/packages.go # const Helm2Version = "2.17.0"
make build
cd ..

Download the last helm

wget https://get.helm.sh/helm-v2.17.0-linux-amd64.tar.gz
tar -xvzf helm-v2.17.0-linux-amd64.tar.gz

Create a new image

echo 'FROM gcr.io/jenkinsxio/builder-go:2.1.150-769
COPY linux-amd64/helm /home/jenkins/.jx/bin/helm
COPY linux-amd64/helm /usr/bin/helm
COPY jx/build/jx /usr/bin/jx' > Dockerfile
docker build . -t builder-go:helm-v2.17.0 $DOCKER_REPO/builder-go:helm-v2.17.0
docker push $DOCKER_REPO/builder-go:helm-v2.17.0

Then use that $DOCKER_REPO/builder-go:helm-v2.17.0 image wherever you need it.

I don't know if all those steps are necessary, but at least they work.

[pod2metra]

All my attempts to make this action work on python or nodejs iamges are failed.
/bin/sh: /usr/bin/jx: cannot execute binary file on setup-jx-git-credentials step

[pod2metra]

I found how to resolve that.

make build should be changed to make linux and COPY jx/build/jx /usr/bin/jx to COPY jx/build/linux/jx /usr/bin/jx

[pod2metra]

@aure-olli may be you know where possible to setup image for env building? I reset image value in configmap jenkins-x-pod-template-go but on env building step it still use old go image gcr.io/jenkinsxio/builder-go:2.1.150-769

[pod2metra]

I found solution. To set image for your env You should overwrite agent in each env repo jenkins-x.yaml.

env:
- name: DEPLOY_NAMESPACE
  value: jx-staging
pipelineConfig:
  agent:
    image: <path to your image>
  env:
  - name: DEPLOY_NAMESPACE
    value: jx-staging
  pipelines: {}

[aure-olli]

Those are currently the fixes I had to perform (with $DOCKER_REPO being your docker repo, gcr.io/my-project on GCE):

dev environment:

In jenkins-x.yaml, replace every

image: builder-go

by

image: $DOCKER_REPO/builder-go:helm-v2.17.0

Other environments:

Add in jenkins-x.yaml

pipelineConfig:
  agent:
    image: $DOCKER_REPO/builder-go:helm-v2.17.0
    label: jenkins-go

Make sure that env/Chart.yaml starts with

apiVersion: v1

For applications

Add in jenkins-x.yaml

pipelineConfig:
  pipelines:
    pullRequest:
      promote:
        preSteps:
        - command: helm init --client-only
          image: asia.gcr.io/olli-iviet/builder-go:helm-v2.17.0-x2
          name: helm-init
    release:
      promote:
        preSteps:
        - command: helm init --client-only
          image: asia.gcr.io/olli-iviet/builder-go:helm-v2.17.0-x2
          name: helm-init

Not sure why needs a helm init, but does.

To create the builder-go:helm-v2.17.0 docker image

Build jx

git clone https://github.com/jenkins-x/jx
cd jx
nano +32 pkg/packages/packages.go # const Helm2Version = "2.17.0"
make linux
cd ..

Download the last helm

wget https://get.helm.sh/helm-v2.17.0-linux-amd64.tar.gz
tar -xvzf helm-v2.17.0-linux-amd64.tar.gz

Create a new image

echo 'FROM gcr.io/jenkinsxio/builder-go:2.1.150-769
COPY linux-amd64/helm /home/jenkins/.jx/bin/helm
COPY linux-amd64/helm /usr/bin/helm
COPY jx/build/linux/jx /usr/bin/jx' > Dockerfile
docker build . -t builder-go:helm-v2.17.0 -t $DOCKER_REPO/builder-go:helm-v2.17.0
docker push $DOCKER_REPO/builder-go:helm-v2.17.0

[aure-olli]

Apparently the team has already update the jx code (image gcr.io/jenkinsxio/builder-go:2.1.153-774).
However, helm is still as old in the image, so I think everything except the dev pipeline will keep failing.

[anmiralles]

Hi everyone, same problem in our project.

We have tried to apply the workaround but this is only valid for dev environment. We have dozens of microservices using different buildpacks and builder images.

We would need jx version 2.1.154 to be pre-released with helm patch applied, and then jenkins-x-versions 1.0.610 (currently 1.0.609) released referencing the first release (jx) updating all builder images.

Any suggestion what we can do? We have all projects stopped.

Thanks in advance.

[aure-olli]

Hi @anmiralles
I gave a work around for all environments and applications. Where does it fail ?

[anmiralles]

Hi @aure-olli ,

your workaround is valid, but it is not applicable if you have dozens of builder images to rebuild and manage in your docker registry....modify all applications pipelines, etc.

Instead of all of these hand customizations, the only thing needed is current version of jx 2.1.154 to be released and jenkins-x-versions pointed to jx version. Moreover, this will come in some days and you will have to disable all you made by hand.

Do you see my point? Maybe I am wrong....

[gcontini]

@anmiralles not wrong...
this is a blocker bug that has been around for days. The workaround is expensive to the point that is unapplicable, and in my company some very bad email start going around. @aure-olli Is there any forecast for when the patched version of jenkins-x will be released?

[aure-olli]

@anmiralles @gcontini
I'm only member because I do many PR so they gave me the privilege to test my PR.
I have no impact on the development process, and I'm as eager as you all to see a general fix coming.

[marziman]

@jstrachan @rawlingsj hi guys,
hope you are well. We just also run into this one and it defacto blocked all of our builds on envs, incl. Production.
Is there any possibility to get this solved? (We could not update fast enought to jx3 alpha).

We can not build everywhere the custom images and apply the idea above.

Thanks & BR
Mehmet

[rawlingsj]

Just seen this sorry as I'm on PTO. After a quick look from my phone it looks like our v2 cluster needs to have the workaround applied to it's own infra so we can release the fixes.

The helm deprecation was some time ago but I wonder if for some reason the service for the old helm stable repo was taken down this week. Horrible timing if that is the case but again will need to investigate.

I dont know how long I can get on my laptop but will try now and build the custom image with the fix from @aure-olli (MASSIVE THANKS BTW) and see if I can upgrade our cluster and then retrigger the real fix #7598 and hopefully that will get through the release pipelines and into the version stream.

[rawlingsj]

Just manually built and pushed an image with the fix, will go through our infra pipelines and switch to that and try to get the release out but I'm running out of time, will give an update shortly.

[marziman]

@rawlingsj will this be applied to other builders too? Your fix will be only fixing the go builder? How about the Java/Maven ones etc., on our side (and collegues above) it affects all of them by a failing "jx step helm release"

[rawlingsj]

this will enable our pipelines to run and release all the builders with the fix although that will need for the version stream release to go through which may take a few hours and thats provided there's no other issues. I dont have long but will do what I can. For now its probably best to wait for the release rather than use the image above so I might edit the comment above and remove the image name so folks dont get caught in two worlds.

[rawlingsj]

closing the fix from @aure-olli as a duplicate of this #7599 which turns out was merged yesterday, though the release pipeline for that PR failed because our infra didn't have the workaround fix applied. It does now for this release pipeline at least and that is running so lets see if we get a PR on the builders repo

[rawlingsj]

I've run out of time ATM but should be able to check back in later, hopefully the fix will get through the builders image and then PR the version stream in that time.

[rawlingsj]

The builder repo update went through ok but the platform failed, then found the old helm version coming from base image repo. This might be the last I can look at tonight as already vastly pushed the boundaries of family time on xmas eve. When I can I'll check back in on the pipelines and fix up remaining issues, I know it's not a good situation to be in but we really got blindsided by the stable repo service stopping this week as we thought we handled the deprecation before the deadline, evidently a service stopped this week and caused severe impact at one of the worst times of year. I've done what I can so far and it seems close as BDD tests now get further but will need the fix above and potentially more. We will do what we can but to set expectations in will likely be after Christmas + Boxing day now.

[aure-olli]

gcr.io/jenkinsxio/builder-jx:2.1.155-777 has the right helm version now, thanks. However needs now to update https://github.com/jenkins-x/jenkins-x-versions to have everything fixed.

Yeah where I live we don't celebrate much Christmas 😅 helm left us a nice Christmas gift ...

[aure-olli]

Oh the PR is already available actually: jenkins-x/jenkins-x-versions#1098
But the pipeline being broken, will have to wait for it to pass manually.

[anmiralles]

Hi everyone,

any updates on this? I can see @rawlingsj sent a PR to jenkins-x-versions with the desire changes that will solve our problems.

Thanks for the effort....

[rawlingsj]

the platform release has now gone through jenkins-x/jenkins-x-platform#7336 and jenkins-x/jenkins-x-versions#1098 has been updated so lets see how the BDD tests go now

[leandromoreirati]

Hello,
I'm experiencing the same problem, when starting the stage from-build-pack, I get the error:

waiting for stage from-build-pack : container step-promote-helm-release to start...

Showing logs for build leandromoreirati/express-hello/master #1 stage from-build-pack and container step-promote-helm-release
WARNING: No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080
Adding missing Helm repo: storage.googleapis.com https://storage.googleapis.com/chartmuseum.jenkins-x.io
error: failed to build dependencies for chart from directory '.': failed to add chart repositories: failed to add the Helm repository with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to add the repository 'storage.googleapis.com' with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to run 'helm repo add storage.googleapis.com https://storage.googleapis.com/chartmuseum.jenkins-x.io' command in directory '.', output: 'Error: Couldn't load repositories file (/tekton/home/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)'

Pipeline failed on stage 'from-build-pack' : container 'step-promote-helm-release'. The execution of the pipeline has stopped.

Hello,
I'm experiencing the same problem, when starting the stage from-build-pack, I get the error:

Follow the full log:
https://gist.github.com/leandromoreirati/19461977b05b3af92f82e5c2d066992c

[rawlingsj]

yeah that's the same error as the version stream BDD tests, this fixes that error jenkins-x/jenkins-x-builders-base@d50e8b1 and is the fix I mentioned that is going through the release pipelines now. The next run of BDD tests are close to being triggered with that fix in so fingers crossed its close now.

[rawlingsj]

another BDD test failure, this should fix it jenkins-x/default-environment-charts@cf9c3d4 but will mean users will need to add the apiVersion to their own environment git repo Chart.yaml when upgrading.

[rawlingsj]

just a quick update that the BDD tests on the version stream PR have started passing jenkins-x/jenkins-x-versions#1098, we have just retriggered the last two that were failing and I'm fairly confident they should pass. Provided they do pass then the PR should be ready to merge in ~1 hour.

[rawlingsj]

jenkins-x/jenkins-x-versions#1098 has merged now, this should fix the issue but please re-open if not.
/close

[jenkins-x-bot]

@rawlingsj: Closing this issue.

In response to this:

jenkins-x/jenkins-x-versions#1098 has merged now, this should fix the issue but please re-open if not.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[marziman]

@rawlingsj I applied the jx binary update. I guess we also need to update the platform. Maybe it makes sense to describe the needed steps to get this fully solved in our CI/CD? Many Thanks.

[ubermensch01]

@marziman I suppose we gotta manage on our own lol.
I upgraded my cluster the following way:

1. navigated to the dev cluster repo folder
2. and ran jx upgrade boot -d .
3. then after a PR was created in that repo I approved it and merged it myself
4. then ran jx boot
5. then I checked whether it helps by re-running a previously failing pipeline.
   Assuming JX is a tender flower, it would make sense to specify a particular version, I am not really sure how to specify that, I guess it would be the --upgrade-version-stream-ref='master': a version stream ref to use to upgrade to argument.
   Anyone care to contribute how to verify gitops version in a cluster better?

[aure-olli]

Apparently the dev environment will fail, a chicken and egg problem.
This is what I had to do

Step 1
Update jx-requirements.yml set versionStream.ref to v1.0.610
update jenkins-x.yml set any agent.image to gcr.io/jenkinsxio/builder-go:2.1.155-778
Commit, push and check that the pipeline succeeds

Step 2
update jenkins-x.yml set back any agent.image to gcr.io/jenkinsxio/builder-go
Commit, push and check that the pipeline succeeds