helm upgrade does not re-create namespace when it is gone

[peay]

When the namespace of a release had been deleted, upgrade fails:

Error: UPGRADE FAILED: failed to create resource: namespaces "xxxxx" not found

This is arguably a special situation, but I would expect helm to re-create the "resources that do not already exist" (https://github.com/kubernetes/helm/blob/master/pkg/kube/client.go#L201), including namespaces.

How about calling ensureNamespace there in Update, as is done in Create? I can try and send a PR for this.

[technosophos]

Is the namespace explicitly declared in the chart?

[peay]

Good point, it is not. This is still slightly inconsistent with the behavior of install though, but that is a good workaround.

[technosophos]

You're right. Just trying to figure out how best to re-create the problem. Thanks!

[peay]

Actually, I am wondering if including the namespace in the chart necessarily works: is it guaranteed to be applied first? I guess if it is not, we could still get the issue above.

[sheerun]

The same for me helm upgrade -i RELEASE CHART --namespace XXX throws "Error: UPGRADE FAILED: failed to create resource: namespaces "XXX" not found" even though helm install --name RELEASE CHART --namespace XXX succeeds. It is indeed inconsistent.

[sheerun]

Another issue, when I crated namespace object like so:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}

and executed helm upgrade -i RELEASE CHART --namespace XXX, I get following error:

Release "RELEASE" does not exist. Installing it now.
Error: release RELEASE failed: namespaces "XXX" already exists

Only the second time I run this command, deployment succeeds..

[sheerun]

My workaround: Add namespace: {{ .Values.namespace }} to each resource and don't use --namespace flag

In helm list it looks like it's deployed under default namespace though.

@technosophos Isn't it weird that even Helm is supposed to work on global level, there are "namespaces" as first-class concept in both cli and helm list output? Wouldn't it make more sense to drop namespace concept from helm cli at all and allow to override namespaces of all resources in some way in templates? Like:

override:
  metadata:
    namespace: {{ .Values.namespace }}

[jayme-github]

/cc @tback @wjkohnen

[nambrot]

Just wanted to say that I also just ran into this. I expected helm upgrade --namespace namespace to install my charts into the specific namespace just like install does.

Note that in my situation, the namespace already exists and it still doesnt install in the namespace

[ptagr]

+1 just ran into this just now!

[Morriz]

yeah...why is it not using the specified namespace on upgrade? anybody can explain?

[ultimateboy]

The easiest way I know how to reproduce the issue is:

$ helm install stable/mongodb --name=mongodb --namespace=mongodb
$ helm delete mongodb
$ kubectl delete ns mongodb
$ helm upgrade --install mongodb stable/mongodb --set replicaCount=2
Error: UPGRADE FAILED: failed to create resource: namespaces "mongodb" not found

[technosophos]

Likely, it is querying the namespace to prepare diffs, and the namespace doesn't exist. So no resource diffs can be generated. And it dies with an error.

On an install, we don't have to perform any queries on a namespace until after the namespace is created.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[waseem-h]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olemarkus]

Just experienced the same issue.

[metral]

/remove-lifecycle stale

[mmack]

Same issue here... It makes helm practically unusable if one tries to create namespaces and default resources (like default ingress per NS, default http backends) for "customers". If you add a new NS to the config - the upgrade fails, so does a install. Kinda stalled there...

[rahulakash]

Hi,

We are still seeing this issue with Helm v2.9.1. This can be reproduced like @ultimateboy has mentioned above.

Thank you.

[bacongobbler]

re-labeling this as a feature request, since we've never supported re-creating the namespace on a helm upgrade. We are busy working on Helm 3 as well as general maintenance on Helm 2, but if someone wants to work on this, contributions are welcome!

[rahulakash]

Thank you @bacongobbler. Just for clarification, I understand if helm upgrade does not support re-creating namespaces. But, we use the --install flag along with upgrade by default helm upgrade --install and it would be nice if namespaces are also re-created in that case (when --install flag is used).

[technosophos]

Oh, that is interesting... I thought --install walked the entire installation process if Tiller detected that the given chart was a new one. Or is this only happening when the release walks the upgrade path?

[mathieu-benoit]

FYI, to be able to reproduce easily the original issue When the namespace of a release had been deleted, upgrade fails:

$ helm install stable/mongodb --name=mongodb --namespace=mongodb
$ kubectl delete ns mongodb
$ helm upgrade --install mongodb stable/mongodb --namespace=mongodb
Error: UPGRADE FAILED: failed to create resource: namespaces "mongodb" not found

Note: same error message without the --install.

[mathieu-benoit]

@ultimateboy, I tried yours explained here #2013 (comment), but you are missing a --purge for the helm delete and a --namespace=mongodb for the helm upgrade. Other than that, I was able to make your script working. Thoughts?

[mathieu-benoit]

FYI, I just opened an associated PR #4600, but still need some guidance around this open question:
Do we have to take into account the --install parameter? Or do that for any helm upgrade if the --namespace is not empty? Or both?

[RehanSaeed]

Is it recommended practice to add a namespace.yml file like so:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}

[aelbarkani]

any updates about this bug ?

[2ZZ]

also seeing this issue

[luisdavim]

helm upgrade --install should have the same behaviour as helm install and create the namespace if missing. So I think this is actually a bug and not a feature request.

[bacongobbler]

For context, helm upgrade --install just checks the release history for an existing release. If it's not present, perform a helm install, otherwise do a helm upgrade. Hope that helps clear things up.

In the install case, helm upgrade --install will create the namespace as expected. The issue here is the problem where helm upgrade will not re-create the namespace if it was deleted out-of-band. It applies to both helm upgrade and helm upgrade --install because the code paths are identical.

It is a feature request; helm upgrade has never re-created the namespace if it was deleted out-of-band. This is a feature request for Helm to re-create resources from scratch if they were purged with kubectl delete.

[luisdavim]

In my experience that is not what happens. Today I was using helm upgrade --install and got an error saying the namespace doesn't exist. This was a new installation without previous releases.

[luisdavim]

any updates on this?

[bacongobbler]

Closing. Helm 3 no longer creates the namespace as part of helm install (see #5753 for background context on why we changed this behaviour - this issue being one of those reasons). Thanks.