Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facilitate ConfigMap rollouts / management #22368

Open
bgrant0607 opened this issue Mar 2, 2016 · 251 comments
Open

Facilitate ConfigMap rollouts / management #22368

bgrant0607 opened this issue Mar 2, 2016 · 251 comments
Labels
area/app-lifecycle area/configmap-api area/declarative-configuration lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/service-catalog Categorizes an issue or PR as relevant to SIG Service Catalog.
Projects

Comments

@bgrant0607
Copy link
Member

bgrant0607 commented Mar 2, 2016

To do a rolling update of a ConfigMap, the user needs to create a new ConfigMap, update a Deployment to refer to it, and delete the old ConfigMap once no pods are using it. This is similar to the orchestration Deployment does for ReplicaSets.

One solution could be to add a ConfigMap template to Deployment and do the management there.

Another could be to support garbage collection of unused ConfigMaps, which is the hard part. That would be useful for Secrets and maybe other objects, also.

cc @kubernetes/sig-apps-feature-requests

@bgrant0607 bgrant0607 added priority/backlog Higher priority than priority/awaiting-more-evidence. area/app-lifecycle team/ux labels Mar 2, 2016
@bgrant0607
Copy link
Member Author

cc @pmorie

@thockin
Copy link
Member

thockin commented Mar 23, 2016

This is one approach. I still want to write a demo, using the live-update
feature of configmap volumes to do rollouts without restarts. It's a
little scarier, but I do think it's useful.
On Mar 2, 2016 9:26 AM, "Brian Grant" notifications@github.com wrote:

To do a rolling update of a ConfigMap, the user needs to create a new
ConfigMap, update a Deployment to refer to it, and delete the old ConfigMap
once no pods are using it. This is similar to the orchestration Deployment
does for ReplicaSets.

One solution could be to add a ConfigMap template to Deployment and do the
management there.

Another could be to support garbage collection of unused ConfigMaps, which
is the hard part. That would be useful for Secrets and maybe other objects,
also.

cc @kubernetes/sig-config
https://github.com/orgs/kubernetes/teams/sig-config


Reply to this email directly or view it on GitHub
#22368.

@bgrant0607 bgrant0607 added this to the next-candidate milestone Mar 23, 2016
@bgrant0607
Copy link
Member Author

@thockin Live update is a different use case than what's discussed here.

@therc
Copy link
Member

therc commented Mar 23, 2016

I think live updates without restarts might fall under my issue, #20200.

@bgrant0607
Copy link
Member Author

@caesarxuchao @lavalamp: We should consider this issue as part of implementing cascading deletion.

@bgrant0607
Copy link
Member Author

Ref #9043 re. in-place rolling updates.

@lavalamp
Copy link
Member

Yeah I think it should be trivial to set a parent for a config map so it automatically gets cleaned up.

(Why not just add a configmap template section to deployment anyway? Seems like a super common thing people will want to do.)

@caesarxuchao
Copy link
Member

@lavalamp, I guess you mean we can set replicas sets as the parent of a config map, and delete the config map when all the replica sets are deleted?

@bgrant0607
Copy link
Member Author

@caesarxuchao Yes

@bgrant0607
Copy link
Member Author

@0xmichalis
Copy link
Contributor

Recent discussion:
https://groups.google.com/forum/#!topic/google-containers/-em3So0KBnA

Thinking out loud: In OpenShift we have the concept of triggers. For example when an image tag is referenced by a DeploymentConfig and there is a new image for that tag, we detect it via a controller loop and update the DeploymentConfig by resolving the tag to the full spec of the image (thus triggering a new deployment since it's a template change). Could we possibly do something similar here? A controller loop watches for configmap changes and triggers a new deployment (we would also need to support redeployments of the same thing since there is no actual template change involved - maybe by adding an annotation to the podtemplate?)

@bgrant0607
Copy link
Member Author

Fundamentally, there need to be multiple ConfigMap objects if we're going to have some pods referring to the new one and others referring to the old one(s), just as with ReplicaSets.

@rata
Copy link
Member

rata commented Mar 30, 2016

On Wed, Mar 30, 2016 at 01:56:24AM -0700, Michail Kargakis wrote:

Recent discussion:
https://groups.google.com/forum/#!topic/google-containers/-em3So0KBnA

Thinking out loud: In OpenShift we have the concept of triggers. For example when an image tag is referenced by a DeploymentConfig and there is a new image for that tag, we detect it via a controller loop and update the DeploymentConfig by resolving the tag to the full spec of the image (thus triggering a new deployment since it's a template change). Could we possibly do something similar here? A controller loop watches for configmap changes and triggers a new deployment (we would also need to support redeployments of the same thing since there is no actual template change involved - maybe by adding an annotation to the podtemplate?)

(I posted the original mail in the thread on the google group)

I think making a deployment is the best way, too. Because you can have a syntax
error or whatever in the config and the new nodes hopefully won't start and the
deployment can be rolled back (or, in not common cases I suspect, even do a
canary deployment of a config change).

But I'm not sure if the configmap should be updated, as you propose, or if it
should be a different one (for kube internals, at least). As, in case you do a
config update with a syntax error a pod will be taken down during the
deployment, a new up that fail and now there is no easy way to rollback because
the configmap has been updated. So, probably you need to update again the
configmap and do another deploy. If it is a different configmap, IIUC, the
rollback can be done easily.

@rata
Copy link
Member

rata commented Apr 1, 2016

Sorry to bother again, but can this be tagged for milestone v1.3 and, maybe, a lower priority?

@rata
Copy link
Member

rata commented Apr 6, 2016

@bgrant0607 ping?

@thockin
Copy link
Member

thockin commented Apr 6, 2016

What work is needed, if we agree deployment is the best path ?

On Tue, Apr 5, 2016 at 5:10 PM, rata notifications@github.com wrote:

@bgrant0607 https://github.com/bgrant0607 ping?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#22368 (comment)

@bgrant0607 bgrant0607 modified the milestones: v1.3, next-candidate Apr 6, 2016
@bgrant0607
Copy link
Member Author

@rata Sorry, I get zillions of notifications every day. Are you volunteering to help with the implementation?

@thockin We need to ensure that the parent/owner on ConfigMap is set to the referencing ReplicaSet(s) when we implement cascading deletion / GC.

@rata
Copy link
Member

rata commented Apr 6, 2016

@bgrant0607 no problem! I can help, yes. Not sure I can get time from my work and I'm quite busy with university, but I'd love to help and probably can find some time. I've never really dealt with kube code (I did a very simple patch only), but I'd love to do it :)

Also, I guess that a ConfigMap can have several owners, right? I think right now it can be used in several RSs and that should be taken into account when doing the cascade deletion/GC (although maybe is something obvious).

Any pointers on where to start? Is someone willing to help with this?

PS: @bgrant0607 sorry the delay, it was midnight here when I got your answer :)

@pmorie
Copy link
Member

pmorie commented Apr 6, 2016

@rata

But I'm not sure if the configmap should be updated, as you propose, or if it should be a different one (for kube internals, at least).

If we manage kube internals with deployments, we have to find the right thing to do for both user-consumed configs and internals.

Also, I guess that a ConfigMap can have several owners, right?

@bgrant0607 I also have the same Q here -- I think we will need to reference-count configmaps / secrets since they can be referred to from pods owned by multiple different controllers.

I think right now it can be used in several RSs and that should be taken into account when doing the cascade deletion/GC (although maybe is something obvious).

Cascading deletion has its own issues: #23656 and #19054

@caesarxuchao
Copy link
Member

@rata, I'm working on cascading deletion and am putting together a PR that adds the necessary API, including the "OwnerReferences". I'll cc you there.

@rata
Copy link
Member

rata commented Apr 6, 2016

@caesarxuchao thanks!

@rata
Copy link
Member

rata commented Apr 6, 2016

On Wed, Apr 06, 2016 at 09:37:25AM -0700, Paul Morie wrote:

@rata

But I'm not sure if the configmap should be updated, as you propose, or if it should be a different one (for kube internals, at least).

If we manage kube internals with deployments, we have to find the right thing to do for both user-consumed configs and internals.

Sure, but I guess the same should probably work for boths, right?

I imagine for example the "internals" configMaps to use a name like
-v<kube-version/commit hash>.

This way, when you upgrade the configmap will became orphan and it should be
deleted, right? Or am I missing something?

I think this can work for boths

I think right now it can be used in several RSs and that should be taken into account when doing the cascade deletion/GC (although maybe is something obvious).

Cascading deletion has its own issues: #23656 and #19054

Ohh, thanks!

@bgrant0607
Copy link
Member Author

@caesarxuchao @rata We'll likely need a custom mechanism in Deployment to ensure that a referenced ConfigMap is owned by the generated ReplicaSets that reference it.

@kfox1111
Copy link

I attended the last contributor summit and there was a discussion about apply --prune. https://www.kubernetes.dev/events/2022/kcsna/schedule/ talk "kubectl apply/prune"

I wouldn't recommend it. Its never made it out of alpha and its sounding increasingly hard to maintain.

@msrb
Copy link

msrb commented Nov 22, 2022

@kfox1111 Thanks for sharing the information!

Here's a link to the google doc from the session: https://docs.google.com/document/d/1y747qL3GYMDDYHR8SqJ-6iWjQEdCJwK_n0z_KHcwa3I/edit#

I have to say that I've never noticed any issues with apply --prune, but that's probably because my use cases were usually quite straight-forward: just deploy an app in a single namespace and then update it.

@christopherglennreed
Copy link

Any Update?

@kfox1111
Copy link

no. It probably needs a developer with time, to express some interest in solving the problem.

@sftim
Copy link
Contributor

sftim commented Dec 22, 2022

For this, could we provide an in-project but out-of-tree CustomResourceDefinition and controller that helps with this story? If it has to be part of core Kubernetes, let's capture why it has to be part of core Kubernetes.

@bgrant0607
Copy link
Member Author

This is a problem that trips up pretty much every user of ConfigMap with any workload controller: Deployment, StatefulSet, DaemonSet, ... It is unfortunate that we haven't been able to address it.

@pentago
Copy link

pentago commented Dec 22, 2022

Ultimately, when a configmap/secret changes, it's rational to expect that pods will reload to propagate the latest change instead of relying on a sea of 3rd party tools for this feature. It's also reasonable to expect that this functionality should be core Kubernetes functionality.

To illustrate how bad the current situation is:
imagine buying a new expensive car that drives great but doesn't have a gas tank and instead you need to go to the repair shop to drill in the hole to put the gas in.

It's a missing, long overdue quality-of-life functionality that the ecosystem deserves.

Sure, we can live without it but we lived without Kubernetes before it as well and got by with no fuss.
Some things are implied, I think, and are common sense. :)

@sbezverk
Copy link
Contributor

sbezverk commented Dec 22, 2022 via email

@thockin
Copy link
Member

thockin commented Dec 22, 2022 via email

@kfox1111
Copy link

@thockin: I did one years ago: kubernetes/enhancements#948

Its still valid I think.

Reviews welcome.

@adampl
Copy link

adampl commented Dec 22, 2022

Ultimately, when a configmap/secret changes, it's rational to expect that pods will reload to propagate the latest change

AfAIK if configmap/secret is mounted as a volume, the changes are immediately reflected in the pods that use them. The problem only concerns pods that read them as environment variables, or mount as volume with subPath (no propagation), or with some types of software running in the pods. Some programs have to be restarted to reload their config, others need just a signal like HUP, and some just read always straight from the file and don't need anything.

So we cannot assume that all pods should restart just because a configmap/secret has changed, because it might break many valid use cases. There should be an option to choose the type of reaction we expect, with the current behavior as the default.

@kfox1111
Copy link

please read the kep. There are use cases where it does make sense to slide in a change at runtime (tls certs for example) But there are also a large class of use cases where its not the right thing to do. The kep covers the need for features around the latter.

The behavior where configmap/secret changes automatically affect pods is an antipattern for a lot of use cases I believe. You want all instances of a replicaset to behave identically. Without the configmaps/secrets being atomic, there is no way to guarantee that during the lifecycle of the replicaset. This issue is about adding the needed functionality to easily be able to guarantee consistency within replicassets while ensuring changes go through the rollout process.

@sftim
Copy link
Contributor

sftim commented Dec 22, 2022

@kfox1111 if you update the KEP issue description to mention this issue, then (IMO) we can close this one in favor of the KEP.

@sftim
Copy link
Contributor

sftim commented Dec 22, 2022

You'll need a KEP issue, not just a PR.

@kfox1111
Copy link

kfox1111 commented Dec 22, 2022

One concrete example.

I have configmap 'hello' with:
setting=1

and deployment hello that references configmap 'hello' with replicas 3.

I check them into git and gitops deploys it.

I then need to upgrade to version 2 of my app. It requires other_setting=2 be set in the configmap. so I upload:
setting=1
other_setting=2
and update the image in the deployment to image:v2

I check it into git, and git

The rollout starts. Say there is a problem rolling out. we now have 2 old pods and 1 new broken pod.

now, say there is a problem on one of the nodes and one of the old pods gets recreated. (hardware failure, node autoupgrade, autoscaling, etc)

it gets recreated with the old deployment spec, but with the new configmap. It doesn't know what other_setting=2 and if we're lucky, it just crashes. In the worst case, it could behave very badly, possibly damaging the system (maybe the configmap hosts bash scripts that it executes poorly because it wasnt designed for it)

Instead of having 2 reversions of the software as intended, there are 3 reversions. the old, the new, and an unintended hybrid of old and new. Its hard to test this, and it cant be guaranteed to not happen.

Its hard to even determine when it happens. As you can't look at the replicaset and know.

This can be manually orchestrated around, but its difficult and isn't gitops friendly today. (can use immutable configmaps with special naming, and owner references, but how do you get an owner reference for the replicaset when your just doing deployment changes? likely need to manually manage your replicasets too...)

We need to do for configmap/secrets what we did for replicasets. wrap it in enough orchestration so the user just uploads desired state and the orchestrator deals with the versioning/snapshotting/consistency in the right way so the user doesn't have to.

@kfox1111
Copy link

@sftim ah. I was unaware of the requirement. I'll file one. Thanks.

@kfox1111
Copy link

This issue, and the kep predates a lot of the new machinery around keps/issues, and I really don't know what I'm doing regarding it all. So, I'll make mistakes (sorry in advance) and welcome assistance. :)

@adampl
Copy link

adampl commented Dec 22, 2022

@kfox1111 I think the problem is with defining what constitutes a pod configuration, and what an app configuration.

The problem you describe also applies to databases and files and anything that has a modifiable state affecting the application. So we could go on and demand that Kubernetes also should apply & rollback file changes and database migrations ;)

IMHO the best solution is to develop apps that have backward and forward compatible configs. I can imagine that having two concurrent app versions (old and new) running on two different configs (two different states) can also lead to nightmarish scenarios.

@kfox1111
Copy link

I understand that some software can not be rolled back. That is unfortunate reality too. But there is a lot of software that is just fine with it, but still suffer from this issue.

https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
"A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods."

Regarding configmaps/secrets with deployments though, its not able to provide identical Pods. This can be fixed. I don't think we should keep looking the other way on it. Its a common enough problem most higher level tools are working around it themselves. And its painful to have to describe to users during training sessions.

@thockin
Copy link
Member

thockin commented Dec 22, 2022

@kfox1111 I will take a look at the KEP - I had not seen it before.

@kfox1111
Copy link

probably needs some revisions and dusting off. but a skim / sniff test would be good.

@Fran-Rg
Copy link

Fran-Rg commented Feb 7, 2023

It's 2023 and still an open problem, let's celebrate.

I found this article, https://blog.questionable.services/article/kubernetes-deployments-configmap-change/ , that was pointing to using config map hash to figure out if a deployment needs to be redeployed on config map change. While it's good it does require something to re-apply the deployment regularly no?

Has no one built a tool that would do all these things for us e.g.

  1. run in the cluster and monitor all config maps
  2. save the config map hash as annotation on all config maps
  3. if found a config map has change, rollout all deployment/job/stateful that use that config

A bit in the spirit of https://codeberg.org/hjacobs/kube-downscaler

@remram44
Copy link

remram44 commented Feb 7, 2023

Not all deployments need to be restarted for configmap changes, some will re-read the file when kubelet updates it.

Also note that updating a configmap in-place doesn't work great with rolling upgrades, e.g. you can't prevent restarting pods from upgrading out-of-turn, and you can't automatically roll back.

@dudicoco
Copy link

dudicoco commented Feb 7, 2023

@Fran-Rg https://github.com/stakater/Reloader/ is the tool you're looking for, but as @remram44 stated it is not a safe mechanism for rolling updates as it is best practice to keep your pods immutable.

We're using a hash with helm and it works great, you don't even need to think of this problem anymore after you've implemented this solution within the chart.

@hedgss
Copy link

hedgss commented Feb 7, 2023

@dudicoco
We also using it in our k8s clusters. It works properly. It is a good workaround.

@adrian-gierakowski
Copy link

kustomize (built into kubectl) will also hash the config map, add the hash to it’s name and replace references to it across the project. Spinnaker has been doing the same thing for years.

@sbrunner
Copy link

sbrunner commented Feb 7, 2023

The hash is not always possible, we sometimes use a ConfigMap or Secret created out of the chart, currently we use The reloader operator as a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/app-lifecycle area/configmap-api area/declarative-configuration lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/service-catalog Categorizes an issue or PR as relevant to SIG Service Catalog.
Projects
Workloads
  
In Progress
Development

No branches or pull requests