Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm3 doesn't automatically create namespace #891

Open
costimuraru opened this issue Oct 9, 2019 · 20 comments
Open

Helm3 doesn't automatically create namespace #891

costimuraru opened this issue Oct 9, 2019 · 20 comments

Comments

@costimuraru
Copy link

@costimuraru costimuraru commented Oct 9, 2019

Helm3 doesn't automatically create namespace - see https://v3.helm.sh/docs/faq/#automatically-creating-namespaces

How can we solve this with helmfile, so that we don't have to manually create them?

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Oct 9, 2019

@costimuraru Good point!

I think incubator/raw would be handy for this use-case:

releases:
- name: namespaces
  chart: incubator/raw
  values:
  - resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: myns1
      spec:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: myns2
      spec:
@costimuraru

This comment has been minimized.

Copy link
Author

@costimuraru costimuraru commented Oct 9, 2019

Thanks, @mumoshu. Yes that also crossed my mind, though we're seeing some issues with incubator/raw and helm3. Opened an issue in helm to track it: helm/helm#6626

@mikesplain

This comment has been minimized.

Copy link
Contributor

@mikesplain mikesplain commented Oct 10, 2019

We've been doing this with hooks for the moment:

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: system-dashboard

#helmfile.yaml
releases:
- name: kubernetes-dashboard
  namespace: system-dashboard
  chart: helm/kubernetes-dashboard
  version: 2.0.0-internal.0
  values:
  - values.yaml.gotmpl
  hooks:
  - events: ["prepare"]
    showlogs: true
    command: "kubectl"
    args: ["apply", "-f", "namespace.yaml"]

You can definitely improve on that but for now that's what we've been doing.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Oct 10, 2019

@mikesplain That's neat. Thanks for sharing!

@aegershman

This comment has been minimized.

Copy link
Contributor

@aegershman aegershman commented Oct 10, 2019

@mikesplain I do something similar for cert-manager; something to mind is that the kubectl command in the hook isn't guaranteed to be the same as the kubeContext declared in the helmfile.yml; it's whatever the current kubectl context of the user's env is. This could make a difference in environments which use helmfile for multiple environments & the user forgets to kubectl config use-context into the cluster their helmfile is targeting. It's a bit of an edge-case but it has implications. With that said, so far using the hook has worked fine. I'm just careful about which kubectl context I'm in 👍

@costimuraru

This comment has been minimized.

Copy link
Author

@costimuraru costimuraru commented Oct 10, 2019

We've also discussed a couple more solutions we could solve this:

  1. Have a create-namespaces-chart which takes a list of namespaces and generates the yaml files for it.
releases:
- name: "create-namespaces-chart"
  chart: "adobe/create-namespaces-chart"
  - name: "namespaces[0]"
    value: "logging"
  - name: "namespaces[1]"
    value: "monitoring"
  - name: "namespaces[2]"
    value: "velero"
  1. Update the upstream charts to create the namespace themselves
@volym3ad

This comment has been minimized.

Copy link

@volym3ad volym3ad commented Oct 23, 2019

We've also discussed a couple more solutions we could solve this:

  1. Have a create-namespaces-chart which takes a list of namespaces and generates the yaml files for it.
releases:
- name: "create-namespaces-chart"
  chart: "adobe/create-namespaces-chart"
  - name: "namespaces[0]"
    value: "logging"
  - name: "namespaces[1]"
    value: "monitoring"
  - name: "namespaces[2]"
    value: "velero"
  1. Update the upstream charts to create the namespace themselves

@costimuraru I don't think this is a good way to create namespaces. In case you accidentally delete chart, all your work will be gone 😱 And eventually the process of recovering all your apps will be a nightmare..

@scottrigby

This comment has been minimized.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Nov 1, 2019

I don't think this is a good way to create namespaces. In case you accidentally delete chart, all your work will be gone

@volym3ad Good point. Definitely this needs to be handled beforehand, or directly by helmfile.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Nov 1, 2019

@scottrigby Thanks for sharing! I'll definitely check helm-namespace and grab the idea.

@mikesplain

This comment has been minimized.

Copy link
Contributor

@mikesplain mikesplain commented Nov 2, 2019

Using a pre-install and pre-upgrade hook can also prevent the deletion problem

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Nov 3, 2019

@mikesplain Good point! Then it seems to be the best option we can have today.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Nov 3, 2019

My immediate idea was to chain any helm calls from Helmfile with helm namespace that seems like an overkill.

The next idea was to change Helmfile to call kubectl or client-go to create the namespace. But it wouldn't work for users who wants to manage namespaces externally to the main helmfile.yaml.

How about adding a flag to optionally enable creating K8s namespace with kubectl? It should look like helmfile apply --create-namespaces. I'd choose shelling out to kubectl as introducing the huge client-go dependency(and conflating the binary size) doesn't pay.

@costimuraru

This comment has been minimized.

Copy link
Author

@costimuraru costimuraru commented Nov 4, 2019

How about adding a flag to optionally enable creating K8s namespace with kubectl? It should look like helmfile apply --create-namespaces.

That could work

@maver1ck

This comment has been minimized.

Copy link

@maver1ck maver1ck commented Dec 13, 2019

Please add this.
incubator/raw have one drawback
helmfile delete deletes is first which deletes all other releases but not all the resources.
This makes a big mess on your cluster.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

@mumoshu mumoshu commented Dec 13, 2019

@maver1ck Which feature do you want specifically and how helmfile delete should work after that?

@hajdukd

This comment has been minimized.

Copy link

@hajdukd hajdukd commented Dec 13, 2019

Worst part about that is that the feature was removed in Helm3 without a notice and cli still takes in namespace param... This is very confusing, just learnt about that while moving to helm3...

@dudicoco

This comment has been minimized.

Copy link

@dudicoco dudicoco commented Dec 31, 2019

I don't really like the idea of a prehook with kubectl apply as it has a few drawbacks.
Here is what I ended up doing, similar to what was suggested by @mumoshu:

values:
- namespaces:
  - test1
  - test2

environments:
  {{ .Environment.Name }}:
    values:
    - global/global-values_{{ .Environment.Name }}.yaml

---

{{ $kubecontext := printf "arn:aws:eks:%s:%s:cluster/%s" $.Values.global.region $.Values.global.account $.Values.global.clusterName }}

releases:
- name: namespaces
  chart: devops/namespace/chart
  version: 1.0.0
  kubeContext: {{ $kubecontext }}
  namespace: kube-system
  values:
  - namespaces:
{{ range $key := $.Values.namespaces }}
    - {{ . }}
{{ end }}

namespace.yaml template within the namespace chart:

{{- range $key := .Values.namespaces -}}

---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ . }}
  labels:
    {{- include "default.labels" $ | nindent 4 }}
  annotations:
    helm.sh/resource-policy: keep

{{- end -}}

This will ensure that when you are running helmfile --environment <env_name> --selector namespace=kube-system apply the namespaces will always be created in the correct kube context.
I've added the helm.sh/resource-policy: keep annotation to make sure the namespace is not accidentally deleted by helm.

Let me know what you think!

@TBeijen

This comment has been minimized.

Copy link
Contributor

@TBeijen TBeijen commented Dec 31, 2019

I've added the helm.sh/resource-policy: keep annotation to make sure the namespace is not accidentally deleted by helm.

That would be the missing detail that would make namespaces managed by helm an option!

Will look into it after new year. Until yet hadn't thought of anything better then bash/kubectl glue, that takes the namespaces defined in the helmfile yaml as a starting point and would precede helmfile.

@zloeber

This comment has been minimized.

Copy link

@zloeber zloeber commented Jan 1, 2020

Added a generic helm chart for what @dudicoco covers here. I figured I'd also cover the other mentioned alternatives and put some helmfile specific example code out there. Need to test with helm 2 as well but it should work I'd think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.