Facilitate ConfigMap rollouts / management #22368
Comments
bgrant0607
added
priority/backlog
|
cc @pmorie |
|
This is one approach. I still want to write a demo, using the live-update
|
|
@thockin Live update is a different use case than what's discussed here. |
|
I think live updates without restarts might fall under my issue, #20200. |
|
@caesarxuchao @lavalamp: We should consider this issue as part of implementing cascading deletion. |
|
Ref #9043 re. in-place rolling updates. |
|
Yeah I think it should be trivial to set a parent for a config map so it automatically gets cleaned up. (Why not just add a configmap template section to deployment anyway? Seems like a super common thing people will want to do.) |
|
@lavalamp, I guess you mean we can set replicas sets as the parent of a config map, and delete the config map when all the replica sets are deleted? |
|
@caesarxuchao Yes |
|
Recent discussion: |
Thinking out loud: In OpenShift we have the concept of triggers. For example when an image tag is referenced by a DeploymentConfig and there is a new image for that tag, we detect it via a controller loop and update the DeploymentConfig by resolving the tag to the full spec of the image (thus triggering a new deployment since it's a template change). Could we possibly do something similar here? A controller loop watches for configmap changes and triggers a new deployment (we would also need to support redeployments of the same thing since there is no actual template change involved - maybe by adding an annotation to the podtemplate?) |
|
Fundamentally, there need to be multiple ConfigMap objects if we're going to have some pods referring to the new one and others referring to the old one(s), just as with ReplicaSets. |
|
On Wed, Mar 30, 2016 at 01:56:24AM -0700, Michail Kargakis wrote:
(I posted the original mail in the thread on the google group) I think making a deployment is the best way, too. Because you can have a syntax But I'm not sure if the configmap should be updated, as you propose, or if it |
|
Sorry to bother again, but can this be tagged for milestone v1.3 and, maybe, a lower priority? |
|
@bgrant0607 ping? |
|
What work is needed, if we agree deployment is the best path ? On Tue, Apr 5, 2016 at 5:10 PM, rata notifications@github.com wrote:
|
|
@bgrant0607 no problem! I can help, yes. Not sure I can get time from my work and I'm quite busy with university, but I'd love to help and probably can find some time. I've never really dealt with kube code (I did a very simple patch only), but I'd love to do it :) Also, I guess that a ConfigMap can have several owners, right? I think right now it can be used in several RSs and that should be taken into account when doing the cascade deletion/GC (although maybe is something obvious). Any pointers on where to start? Is someone willing to help with this? PS: @bgrant0607 sorry the delay, it was midnight here when I got your answer :) |
If we manage kube internals with deployments, we have to find the right thing to do for both user-consumed configs and internals.
@bgrant0607 I also have the same Q here -- I think we will need to reference-count configmaps / secrets since they can be referred to from pods owned by multiple different controllers.
|
|
@rata, I'm working on cascading deletion and am putting together a PR that adds the necessary API, including the "OwnerReferences". I'll cc you there. |
|
@caesarxuchao thanks! |
|
On Wed, Apr 06, 2016 at 09:37:25AM -0700, Paul Morie wrote:
Sure, but I guess the same should probably work for boths, right? I imagine for example the "internals" configMaps to use a name like This way, when you upgrade the configmap will became orphan and it should be I think this can work for boths
Ohh, thanks! |
|
@caesarxuchao @rata We'll likely need a custom mechanism in Deployment to ensure that a referenced ConfigMap is owned by the generated ReplicaSets that reference it. |
|
I attended the last contributor summit and there was a discussion about apply --prune. https://www.kubernetes.dev/events/2022/kcsna/schedule/ talk "kubectl apply/prune" I wouldn't recommend it. Its never made it out of alpha and its sounding increasingly hard to maintain. |
|
@kfox1111 Thanks for sharing the information! Here's a link to the google doc from the session: https://docs.google.com/document/d/1y747qL3GYMDDYHR8SqJ-6iWjQEdCJwK_n0z_KHcwa3I/edit# I have to say that I've never noticed any issues with |
|
Any Update? |
|
no. It probably needs a developer with time, to express some interest in solving the problem. |
|
For this, could we provide an in-project but out-of-tree CustomResourceDefinition and controller that helps with this story? If it has to be part of core Kubernetes, let's capture why it has to be part of core Kubernetes. |
|
This is a problem that trips up pretty much every user of ConfigMap with any workload controller: Deployment, StatefulSet, DaemonSet, ... It is unfortunate that we haven't been able to address it. |
|
Ultimately, when a configmap/secret changes, it's rational to expect that pods will reload to propagate the latest change instead of relying on a sea of 3rd party tools for this feature. It's also reasonable to expect that this functionality should be core Kubernetes functionality. To illustrate how bad the current situation is: It's a missing, long overdue quality-of-life functionality that the ecosystem deserves. Sure, we can live without it but we lived without Kubernetes before it as well and got by with no fuss. |
|
Please allow to use this brilliant illustration in future cases
|
|
KEPs accepted.
…On Thu, Dec 22, 2022, 7:52 AM Serguei Bezverkhi ***@***.***> wrote:
Please allow to use this brilliant illustration in future cases
|
|
@thockin: I did one years ago: kubernetes/enhancements#948 Its still valid I think. Reviews welcome. |
AfAIK if configmap/secret is mounted as a volume, the changes are immediately reflected in the pods that use them. The problem only concerns pods that read them as environment variables, or mount as volume with subPath (no propagation), or with some types of software running in the pods. Some programs have to be restarted to reload their config, others need just a signal like HUP, and some just read always straight from the file and don't need anything. So we cannot assume that all pods should restart just because a configmap/secret has changed, because it might break many valid use cases. There should be an option to choose the type of reaction we expect, with the current behavior as the default. |
|
please read the kep. There are use cases where it does make sense to slide in a change at runtime (tls certs for example) But there are also a large class of use cases where its not the right thing to do. The kep covers the need for features around the latter. The behavior where configmap/secret changes automatically affect pods is an antipattern for a lot of use cases I believe. You want all instances of a replicaset to behave identically. Without the configmaps/secrets being atomic, there is no way to guarantee that during the lifecycle of the replicaset. This issue is about adding the needed functionality to easily be able to guarantee consistency within replicassets while ensuring changes go through the rollout process. |
|
@kfox1111 if you update the KEP issue description to mention this issue, then (IMO) we can close this one in favor of the KEP. |
|
You'll need a KEP issue, not just a PR. |
|
One concrete example. I have configmap 'hello' with: and deployment hello that references configmap 'hello' with replicas 3. I check them into git and gitops deploys it. I then need to upgrade to version 2 of my app. It requires other_setting=2 be set in the configmap. so I upload: I check it into git, and git The rollout starts. Say there is a problem rolling out. we now have 2 old pods and 1 new broken pod. now, say there is a problem on one of the nodes and one of the old pods gets recreated. (hardware failure, node autoupgrade, autoscaling, etc) it gets recreated with the old deployment spec, but with the new configmap. It doesn't know what other_setting=2 and if we're lucky, it just crashes. In the worst case, it could behave very badly, possibly damaging the system (maybe the configmap hosts bash scripts that it executes poorly because it wasnt designed for it) Instead of having 2 reversions of the software as intended, there are 3 reversions. the old, the new, and an unintended hybrid of old and new. Its hard to test this, and it cant be guaranteed to not happen. Its hard to even determine when it happens. As you can't look at the replicaset and know. This can be manually orchestrated around, but its difficult and isn't gitops friendly today. (can use immutable configmaps with special naming, and owner references, but how do you get an owner reference for the replicaset when your just doing deployment changes? likely need to manually manage your replicasets too...) We need to do for configmap/secrets what we did for replicasets. wrap it in enough orchestration so the user just uploads desired state and the orchestrator deals with the versioning/snapshotting/consistency in the right way so the user doesn't have to. |
|
@sftim ah. I was unaware of the requirement. I'll file one. Thanks. |
|
This issue, and the kep predates a lot of the new machinery around keps/issues, and I really don't know what I'm doing regarding it all. So, I'll make mistakes (sorry in advance) and welcome assistance. :) |
|
@kfox1111 I think the problem is with defining what constitutes a pod configuration, and what an app configuration. The problem you describe also applies to databases and files and anything that has a modifiable state affecting the application. So we could go on and demand that Kubernetes also should apply & rollback file changes and database migrations ;) IMHO the best solution is to develop apps that have backward and forward compatible configs. I can imagine that having two concurrent app versions (old and new) running on two different configs (two different states) can also lead to nightmarish scenarios. |
|
I understand that some software can not be rolled back. That is unfortunate reality too. But there is a lot of software that is just fine with it, but still suffer from this issue. https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ Regarding configmaps/secrets with deployments though, its not able to provide identical Pods. This can be fixed. I don't think we should keep looking the other way on it. Its a common enough problem most higher level tools are working around it themselves. And its painful to have to describe to users during training sessions. |
|
@kfox1111 I will take a look at the KEP - I had not seen it before. |
|
probably needs some revisions and dusting off. but a skim / sniff test would be good. |
|
It's 2023 and still an open problem, let's celebrate. I found this article, https://blog.questionable.services/article/kubernetes-deployments-configmap-change/ , that was pointing to using config map hash to figure out if a deployment needs to be redeployed on config map change. While it's good it does require something to re-apply the deployment regularly no? Has no one built a tool that would do all these things for us e.g.
A bit in the spirit of https://codeberg.org/hjacobs/kube-downscaler |
|
Not all deployments need to be restarted for configmap changes, some will re-read the file when kubelet updates it. Also note that updating a configmap in-place doesn't work great with rolling upgrades, e.g. you can't prevent restarting pods from upgrading out-of-turn, and you can't automatically roll back. |
|
@Fran-Rg https://github.com/stakater/Reloader/ is the tool you're looking for, but as @remram44 stated it is not a safe mechanism for rolling updates as it is best practice to keep your pods immutable. We're using a hash with helm and it works great, you don't even need to think of this problem anymore after you've implemented this solution within the chart. |
|
@dudicoco |
|
kustomize (built into kubectl) will also hash the config map, add the hash to it’s name and replace references to it across the project. Spinnaker has been doing the same thing for years. |
|
The hash is not always possible, we sometimes use a ConfigMap or Secret created out of the chart, currently we use The reloader operator as a workaround. |
and others
To do a rolling update of a ConfigMap, the user needs to create a new ConfigMap, update a Deployment to refer to it, and delete the old ConfigMap once no pods are using it. This is similar to the orchestration Deployment does for ReplicaSets.
One solution could be to add a ConfigMap template to Deployment and do the management there.
Another could be to support garbage collection of unused ConfigMaps, which is the hard part. That would be useful for Secrets and maybe other objects, also.
cc @kubernetes/sig-apps-feature-requests
The text was updated successfully, but these errors were encountered: