Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to Helm 3.0 #98

Open
armab opened this issue Nov 15, 2019 · 23 comments
Open

Migrate to Helm 3.0 #98

armab opened this issue Nov 15, 2019 · 23 comments

Comments

@armab
Copy link
Member

@armab armab commented Nov 15, 2019

Helm v3.0 GA was released in Nov 2019 https://helm.sh/blog/helm-3-released/

While at the moment we'd prefer to pin Helm 2.x, consider migrating to 3.x when it'll stabilize a little bit and see what involves for us to rely on latest Helm.

@armab armab added the feature label Nov 15, 2019
@armab armab added this to the prod/GA milestone Nov 15, 2019
@armab armab added this to TODO in ST2 HA in Kubernetes via automation Nov 15, 2019
@armab armab mentioned this issue Nov 15, 2019
@armab armab added the Helm label Nov 15, 2019
@rahulparihar270992

This comment has been minimized.

Copy link

@rahulparihar270992 rahulparihar270992 commented Dec 2, 2019

Hi @armab , hope you are doing well ? As you have added migrate to helm 3.0 feature for stackstorm . Do you have any initial thoughts how you are going to take it forward ? If i am correct i have made change in https://github.com/StackStorm/stackstorm-ha/blob/master/.circleci/config.yml#L20 file to point helm version 3.0.0 . but still getting issue as
helm install stackstorm-ha/ --generate-name
Error: found in Chart.yaml, but missing in charts/ directory: rabbitmq-ha, mongodb-replicaset, external-dns, etcd-operator Do you have any pointers for this?

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 2, 2019

These are 3rd party Helm charts stackstorm-ha relies on https://github.com/helm/charts/blob/master/stable/

From the message above, looks like in order to switch to Helm 3.0 those upstream dependencies needs to be migrated first.
See https://github.com/helm/charts/blob/master/stable/rabbitmq-ha/Chart.yaml#L2 which still references previous Helm API Version.

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 2, 2019

Hi @armab, I installed rabbitmq-ha-1.36.4 first with helm 3.0 and it succeeded:

NAMESPACE NAME READY STATUS RESTARTS AGE IP
default my-rabbit-rabbitmq-ha-0 1/1 Running 1 2d8h 10.244.3.29
default my-rabbit-rabbitmq-ha-1 1/1 Running 1 2d8h 10.244.1.24
default my-rabbit-rabbitmq-ha-2 1/1 Running 1 2d8h 10.244.2.14

So does it mean this chart works in helm 3.
But if install ss ha with :
helm install stackstorm -n st2 -f values.yaml .

The error is:

manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "EtcdCluster" in version "etcd.database.coreos.com/v1beta2"

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 2, 2019

That part is coming from the "etcd-operator" dependency from the official Helm charts repository.

Relevant answer from similar kubernetes-sigs/kubefed#1143 (comment):

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 2, 2019

@armab Thanks. So the left is etcd-operator's issue. Let's wait for it to be updated to support helm 3.

@rahulparihar270992

This comment has been minimized.

Copy link

@rahulparihar270992 rahulparihar270992 commented Dec 3, 2019

These are 3rd party Helm charts stackstorm-ha relies on https://github.com/helm/charts/blob/master/stable/

From the message above, looks like in order to switch to Helm 3.0 those upstream dependencies needs to be migrated first.
See https://github.com/helm/charts/blob/master/stable/rabbitmq-ha/Chart.yaml#L2 which still references previous Helm API Version.

Thanks @armab and @wantdrink for replying back . have tried to make changes in https://github.com/rahulparihar270992/charts repo as belows https://github.com/rahulparihar270992/charts/blob/master/stable/rabbitmq-ha/Chart.yaml#L2 . and tried to changed requirements.yml as https://github.com/rahulparihar270992/stackstorm-ha/blob/feature/stackstome-v1/requirements.yaml#L4 . Not sure this is the right pointing to stable dependencies ? Do you have any suggestion how it going to work? Still facing similar issue.
Error: found in Chart.yaml, but missing in charts/ directory: rabbitmq-ha, mongodb-replicaset, external-dns, etcd-operator

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 3, 2019

I hadn't chance to really try Helm 3.0 yet, so google is your better friend than me here.

However looks like the first big visible difference between v2 and v3 is that requirements should be moved to Chart.yml instead of requirements.yml file.

@rahulparihar270992

This comment has been minimized.

Copy link

@rahulparihar270992 rahulparihar270992 commented Dec 5, 2019

Hi @armab, I installed rabbitmq-ha-1.36.4 first with helm 3.0 and it succeeded:

NAMESPACE NAME READY STATUS RESTARTS AGE IP
default my-rabbit-rabbitmq-ha-0 1/1 Running 1 2d8h 10.244.3.29
default my-rabbit-rabbitmq-ha-1 1/1 Running 1 2d8h 10.244.1.24
default my-rabbit-rabbitmq-ha-2 1/1 Running 1 2d8h 10.244.2.14

So does it mean this chart works in helm 3.
But if install ss ha with :
helm install stackstorm -n st2 -f values.yaml .

The error is:

manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "EtcdCluster" in version "etcd.database.coreos.com/v1beta2"

@wantdrink , @armab I am also facing similar issue . Have you updated helm chart for etcd-operator ?
Can you please give me some pointer here, what next step have you taken?

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 9, 2019

@rahulparihar270992 SS chart has a set of dependencies chart. Fortunately some of them already supports Helm3.
unfortunately etcd-operator as one mandatory dependency cannot be omitted and must be installed. But its chart hasn't been updated yet for Helm3.0 (version 0.10.0 in requirement).
Let me try 0.10.2 today and see if it works.

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 9, 2019

@armab Thanks, you are right.

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 9, 2019

0.10.2 can be installed with helm3. It looks like we can move forward.

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 10, 2019

Sounds promising.
Can you guys contribute a PR to address that?

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 12, 2019

Hi @armab I'd like to do that. I'm still testing today with outside rabibtmq & mongo instead of installing them with ss chart.
Now I pods/services/statefulset/ep are created successfully, login portal/cli/default pack action running all good.
But I still have some questions:

  1. There is still "crd-install" warning but it seems ss works well. Could you please guide us for how to validate if etcd cluster works well together with ss? Maybe some simple testing is needed.

  2. I want to put ss images into my private repository, in that way I can deploy my own pack image to test, also to faster the deployment. I added repository in values.yml as below but it doesn't work.

image:
  repository: repository.test.com:8443/stackstorm
  pullPolicy: IfNotPresent

Then I changed templates/_helpers.tpl as:

# Generate Docker image repository: Private 'docker.stackstorm.com' for Enterprise vs Public Docker Hub 'stackstorm' for FOSS version
{{- define "imageRepository" -}}
{{- if required "Missing context '.Values.enterprise.enabled'!" .Values.enterprise.enabled -}}
docker.stackstorm.com
{{- else -}}
repository.test.com:8443/stackstorm
{{- end -}}
{{- end -}}

It didn't work either and images were stilled pulled from docker hub.

  1. The same with item 2 how can I change the default repository of etcd-operator?
    I changed the stackstorm-ha/charts/etcd-operator/values.yaml and it didn't work.
    E.G.
# etcdOperator
etcdOperator:
  name: etcd-operator
  replicaCount: 1
  image:
    repository: repository.test.com:8443/quay.io/coreos/etcd-operator
    tag: v0.9.4
    pullPolicy: IfNotPresent
  resources:
    cpu: 100m
    memory: 128Mi

  1. Is that possible we develop our own pack with python 3.8 and running in this HA?
@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 13, 2019

Some of the StackStorm services will just fail if they couldn't contact etcd for coordination.
For using custom st2 packs, see: https://github.com/StackStorm/stackstorm-ha#install-custom-st2-packs-in-the-cluster and https://docs.stackstorm.com/install/k8s_ha.html#custom-st2-packs.

You can ask other questions in StackStorm Community: https://stackstorm.com/#community Slack or Forum.
Let's please keep this issue dedicated to Helm v3.0.

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 14, 2019

Thanks @armab
Find the root cause. I used tgz file for command helm install and the image repository of ss/etcd cannot be changed by passing the values.yaml.
To solve this just modify the _helper and charts/etcdxxx/values.yml of tgz file then rerun and finally the helm pull images from my private repository.

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 15, 2019

@armab I've tested the installation of etcd-operator 0.10.2. There is an error when executing helm install but everything can be installed successfully.
Also I tested ss feature and it looks good.

Finally I rebooted all k8s nodes and there is an issue: the etcd cluster pods cannot be restarted automatically:

default       etcd-cluster-6255djfdkq                              0/1     Completed          0          17m     <none>          
default       etcd-cluster-k5lmchwjp5                              0/1     Completed          0          16m     <none>          
default       etcd-cluster-nlzjt5qmdn                              0/1     Completed          0          17m     <none> 

I have to restart them manually and after that ss works well.

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Dec 15, 2019

Good to hear.

BTW Etcd issue you mentioning is not really related to Helm 3.0. We have it documented here: #94

@wantdrink

This comment has been minimized.

Copy link

@wantdrink wantdrink commented Dec 15, 2019

@armab hum......maybe single etcd instance is better.
Thanks.

@erenatas

This comment has been minimized.

Copy link
Contributor

@erenatas erenatas commented Dec 18, 2019

Hi @armab, I installed rabbitmq-ha-1.36.4 first with helm 3.0 and it succeeded:

NAMESPACE NAME READY STATUS RESTARTS AGE IP
default my-rabbit-rabbitmq-ha-0 1/1 Running 1 2d8h 10.244.3.29
default my-rabbit-rabbitmq-ha-1 1/1 Running 1 2d8h 10.244.1.24
default my-rabbit-rabbitmq-ha-2 1/1 Running 1 2d8h 10.244.2.14

So does it mean this chart works in helm 3.
But if install ss ha with :
helm install stackstorm -n st2 -f values.yaml .

The error is:

manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "EtcdCluster" in version "etcd.database.coreos.com/v1beta2"

It seems like I am late to this conversation but I have managed to find a workaround to this problem which was to install etcd operator manually before installing stackstorm-ha. Currently I can run Stackstorm without any issues.

But I must say that, Helm's error traces are just awful. It says there is an issue but does not say about where the problem is...

@Fauli

This comment has been minimized.

Copy link

@Fauli Fauli commented Jan 8, 2020

Do you have an eta for the migration of the charts to helm 3?
Or is there some repo available that has the adopted files?
@wantdrink your example with external components would be very nice and helpful.

Thanks a lot!

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Jan 8, 2020

There is no ETA for migrating to Helm 3, but we’ll be happy to accept PR contributions.

@Fauli

This comment has been minimized.

Copy link

@Fauli Fauli commented Jan 9, 2020

Is there some progress to build upon available or does one have to start from the helm 2 charts?

@armab

This comment has been minimized.

Copy link
Member Author

@armab armab commented Jan 9, 2020

You can read this Issue history with the pointers regarding Helm 2 migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.