New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`helm upgrade` does not re-create namespace when it is gone #2013

Open
peay opened this Issue Feb 23, 2017 · 27 comments

Comments

Projects
None yet
@peay
Contributor

peay commented Feb 23, 2017

When the namespace of a release had been deleted, upgrade fails:

Error: UPGRADE FAILED: failed to create resource: namespaces "xxxxx" not found

This is arguably a special situation, but I would expect helm to re-create the "resources that do not already exist" (https://github.com/kubernetes/helm/blob/master/pkg/kube/client.go#L201), including namespaces.

How about calling ensureNamespace there in Update, as is done in Create? I can try and send a PR for this.

@technosophos

This comment has been minimized.

Member

technosophos commented Feb 23, 2017

Is the namespace explicitly declared in the chart?

@peay

This comment has been minimized.

Contributor

peay commented Feb 23, 2017

Good point, it is not. This is still slightly inconsistent with the behavior of install though, but that is a good workaround.

@technosophos

This comment has been minimized.

Member

technosophos commented Feb 23, 2017

You're right. Just trying to figure out how best to re-create the problem. Thanks!

@technosophos technosophos added this to the 2.4.0-Triage milestone Feb 23, 2017

@peay

This comment has been minimized.

Contributor

peay commented Feb 23, 2017

Actually, I am wondering if including the namespace in the chart necessarily works: is it guaranteed to be applied first? I guess if it is not, we could still get the issue above.

@sheerun

This comment has been minimized.

Contributor

sheerun commented Mar 24, 2017

The same for me helm upgrade -i RELEASE CHART --namespace XXX throws "Error: UPGRADE FAILED: failed to create resource: namespaces "XXX" not found" even though helm install --name RELEASE CHART --namespace XXX succeeds. It is indeed inconsistent.

@sheerun

This comment has been minimized.

Contributor

sheerun commented Mar 24, 2017

Another issue, when I crated namespace object like so:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}

and executed helm upgrade -i RELEASE CHART --namespace XXX, I get following error:

Release "RELEASE" does not exist. Installing it now.
Error: release RELEASE failed: namespaces "XXX" already exists

Only the second time I run this command, deployment succeeds..

@sheerun

This comment has been minimized.

Contributor

sheerun commented Mar 24, 2017

My workaround: Add namespace: {{ .Values.namespace }} to each resource and don't use --namespace flag

In helm list it looks like it's deployed under default namespace though.

@technosophos Isn't it weird that even Helm is supposed to work on global level, there are "namespaces" as first-class concept in both cli and helm list output? Wouldn't it make more sense to drop namespace concept from helm cli at all and allow to override namespaces of all resources in some way in templates? Like:

override:
  metadata:
    namespace: {{ .Values.namespace }}
@jayme-github

This comment has been minimized.

jayme-github commented Jun 8, 2017

/cc @tback @wjkohnen

@nambrot

This comment has been minimized.

nambrot commented Aug 15, 2017

Just wanted to say that I also just ran into this. I expected helm upgrade --namespace namespace to install my charts into the specific namespace just like install does.

Note that in my situation, the namespace already exists and it still doesnt install in the namespace

@ptagr

This comment has been minimized.

ptagr commented Sep 7, 2017

+1 just ran into this just now!

@Morriz

This comment has been minimized.

Morriz commented Oct 3, 2017

yeah...why is it not using the specified namespace on upgrade? anybody can explain?

@ultimateboy

This comment has been minimized.

Contributor

ultimateboy commented Oct 9, 2017

The easiest way I know how to reproduce the issue is:

$ helm install stable/mongodb --name=mongodb --namespace=mongodb
$ helm delete mongodb
$ kubectl delete ns mongodb
$ helm upgrade --install mongodb stable/mongodb --set replicaCount=2
Error: UPGRADE FAILED: failed to create resource: namespaces "mongodb" not found
@technosophos

This comment has been minimized.

Member

technosophos commented Oct 9, 2017

Likely, it is querying the namespace to prepare diffs, and the namespace doesn't exist. So no resource diffs can be generated. And it dies with an error.

On an install, we don't have to perform any queries on a namespace until after the namespace is created.

BugRoger added a commit to sapcc/kubernikus that referenced this issue Nov 13, 2017

@fejta-bot

This comment has been minimized.

fejta-bot commented Jan 7, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@waseem-h

This comment has been minimized.

waseem-h commented Jan 12, 2018

/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

fejta-bot commented Apr 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@olemarkus

This comment has been minimized.

olemarkus commented May 8, 2018

Just experienced the same issue.

@metral

This comment has been minimized.

metral commented May 24, 2018

/remove-lifecycle stale

@mmack

This comment has been minimized.

mmack commented May 25, 2018

Same issue here... It makes helm practically unusable if one tries to create namespaces and default resources (like default ingress per NS, default http backends) for "customers". If you add a new NS to the config - the upgrade fails, so does a install. Kinda stalled there...

@rahulakash

This comment has been minimized.

rahulakash commented Aug 1, 2018

Hi,

We are still seeing this issue with Helm v2.9.1. This can be reproduced like @ultimateboy has mentioned above.

Thank you.

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Aug 1, 2018

re-labeling this as a feature request, since we've never supported re-creating the namespace on a helm upgrade. We are busy working on Helm 3 as well as general maintenance on Helm 2, but if someone wants to work on this, contributions are welcome!

@bacongobbler bacongobbler added the feature label Aug 1, 2018

@rahulakash

This comment has been minimized.

rahulakash commented Aug 1, 2018

Thank you @bacongobbler. Just for clarification, I understand if helm upgrade does not support re-creating namespaces. But, we use the --install flag along with upgrade by default helm upgrade --install and it would be nice if namespaces are also re-created in that case (when --install flag is used).

@technosophos

This comment has been minimized.

Member

technosophos commented Aug 2, 2018

Oh, that is interesting... I thought --install walked the entire installation process if Tiller detected that the given chart was a new one. Or is this only happening when the release walks the upgrade path?

@mathieu-benoit

This comment has been minimized.

mathieu-benoit commented Sep 6, 2018

FYI, to be able to reproduce easily the original issue When the namespace of a release had been deleted, upgrade fails:

$ helm install stable/mongodb --name=mongodb --namespace=mongodb
$ kubectl delete ns mongodb
$ helm upgrade --install mongodb stable/mongodb --namespace=mongodb
Error: UPGRADE FAILED: failed to create resource: namespaces "mongodb" not found

Note: same error message without the --install.

@mathieu-benoit

This comment has been minimized.

mathieu-benoit commented Sep 6, 2018

@ultimateboy, I tried yours explained here #2013 (comment), but you are missing a --purge for the helm delete and a --namespace=mongodb for the helm upgrade. Other than that, I was able to make your script working. Thoughts?

@mathieu-benoit

This comment has been minimized.

mathieu-benoit commented Sep 6, 2018

FYI, I just opened an associated PR #4600, but still need some guidance around this open question:
Do we have to take into account the --install parameter? Or do that for any helm upgrade if the --namespace is not empty? Or both?

@RehanSaeed

This comment has been minimized.

RehanSaeed commented Oct 4, 2018

Is it recommended practice to add a namespace.yml file like so:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment