ConfigMaps consuming pod facts

[nebril]

The problem

I would like to use pod facts (such as pod IP address) in ConfigMaps, so that each container configuration is populated with the correct value of a pod fact.

Possible solution

The idea is to add templates to ConfigMap values, so each time a ConfigMap is applied to a pod - it can transform the value into something specific to a pod.

The example syntax would be $(variable.name).

Example case

We assume that there are two possible variables (templates) that can be used to consume pod facts: pod.IP and pod.name, which evaluate to pod.Status.podIP and pod.Name.

Let's consider following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  service.ip: $(pod.IP)
  other.value: trololo
  service.name: $(pod.name)

Then we have a pod with a service that needs to put the IP of the pod in the config file /etc/config/service.ip.

Pod manifest can look like that:

apiVersion: v1
kind: Pod
metadata:
  name: pod-facts-consumer
spec:
  containers:
    - name: test-container
      image: gcr.io/google_containers/busybox
      command: [ "/bin/sh", "cat", "/etc/config/service.ip" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: special-config
  restartPolicy: Never

When we create above ConfigMap and run the pod, we expect the test-container output to be an IP address that was assigned to the Pod.

Proof of concept

The proof of concept implementation is demoed here: https://asciinema.org/a/e90cs3ymgtatdexk6rmjuet92

The code for above demo will be linked to this issue via WiP Pull Request.

Benefits

With this mechanism in place, we can have clean configuration of containers that need to be aware of their IPs. We can drop init containers creating additional volumes mounted in final containers - the state passed to container happens implicitly by templates defined by user, which should add to overall user experience in cases of writing configuration for such applications.

Real world example

An example of such application can be Galera, which needs local IP addresses in my.conf file.

Alternatives

The alternative way to achieve this which already works is using init containers to create volumes filled with data from downward API, which are then mounted in the container running the desired workload.

The downside of above approach requires you to compile pod facts into configuration files using additional scripts or template engines, which add to complexity of pod configuration.

In contrast, if this feature is implemented you can declare configuration files with multiple pod facts in them directly in ConfigMap, which moves the burden of injecting them into config files away from cluster user.

Previous submissions

This document was previously submitted by me here:
kubernetes/enhancements#38
I was pointed out to me that it needs more discussion before submitting a feature.

[jberkus]

Why rename the pod fact variables? Why not just use the canonical downward API names?

[jberkus]

Also: will this interpretation work regardless of where the $(var) is in the configmap? i.e. will it work for yml config data, or with concatination? Examples:

data:
    api.address: $(pod.IP):8001

data:
    api.config: |
          admin_host: $(pod.IP)
          node_name: $(pod.name)
          mode: replication

[pmorie]

IMO, the correct way to do this is to use the downward API. I suggest we create an issue for populating template files from secrets, downward API, config maps, and arbitrary other sources of information if that is a desire. We have discussed templates quite a bit in the past, but I believe this is a new flavor of the templating problem that we haven't previously considered.

See discussion on #27880 -- in this issue, we have some discussion of making the volume plugin and env features of downward API have parity (see also #29644). I would much rather spend the effort on making the downward API volume plugin than introduce unwarranted complexity and a new flavor of downward API syntax in the configmap plugin.

[pmorie]

For the record, I don't debate at all that populating a template with different sources of information from the kube API (secrets, configmap, downward API etc) is a valid problem. That said, that feature would be more than just a change to configmap, and definitely not a change to the configmap volume plugin.

[jberkus]

It's really the templating issue that I'm interested in. The overarching goal is to get rid of entrypoint.sh scripts, without needing to make significant modifications to the containerized applications.

[pmorie]

@jberkus yeah, I am also interested in that too. I look forward to discussing it when we have an issue :)

[nebril]

@jberkus Multiple variables in one ConfigMap data value and getting rid of entrypoint scripts is exactly the goal that this feature is meant to achieve. It actually works already in the PoC (with env vars), I probably should have pointed this out more clearly.

@pmorie your proposal makes a lot of sense, I will investigate older discussions and think about formulating new issue, which will include consuming ConfigMaps, Secrets and Downward API in template files.

[pmorie]

@nebril Also for the record, we have discussed in the past having a 'super volume' that knew how to project the three different types of info into a volume (ConfigMap, Secret, Downward API) into a single volume. What you want sounds like that + a template.

[jberkus]

@pmorie sounds great to me

[nebril]

@pmorie Was there any proposal/feature draft for this 'super volume'? If there was, can you link to it?

[thockin]

I am with Paul on this. I think this is a dramatic abuse of the configmap volume.

[kfox1111]

What about a templatemap type? Similar to config map, but can pull data from volumes associated with the pod like the downward api, secrets, and other configmaps?

[pmorie]

@nebril I'll try to dig the discussion up soon. I think I made a run at coding it, cannot remember exactly.

[yoojinl]

Another important use-case which I see for templates is configuration of credentials for the database. We may have an object which describes the credentials (user and password). The same object should be used for user creation after Galera cluster is installed, and further by template for service which requires the access to the database.

Something like:

data:
    keystone.conf: |
        [database]
        connection = mysql+pymysql://$(keystone_db.username):$(keystone_db.password)@mysql/$(keystone_db.name)

[thockin]

I don't think we want that built-in to Kubernetes. There are so many ways
people can do this and if we bake it in, we will be forever fielding
enhancement requests. Better instead to demonstrate a sidecar pattern.
Pull the configmap into a confd helper (or something else), and let confd
render it into a shared emptydir.

On Aug 10, 2016 4:05 AM, "Evgeniy L" notifications@github.com wrote:

Another important use-case which I see for templates is configuration of
credentials for the database. We may have an object which describes the
credentials (user and password). The same object should be used for user
creation after Galera cluster is installed, and further by template for
service which requires the access to the database.

Something like:

data:
keystone.conf: |
[database]
connection = mysql+pymysql://$(keystone_db.username):$(keystone_db.password)@mysql/$(keystone_db.name)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVPXdic7rjRJByCW77S0vUlWHwO_wks5qebB3gaJpZM4JU_a8
.

[kfox1111]

@thockin: very interesting idea...

You'd want some way to make sure the config's rendered before the other container starts though. Think an init container would work for this?

[kfox1111]

Also, should Kubernetes provide a set of blessed sidecar containers for things like this? K8s already provides one. pause. I'd be more inclined to use one if I knew it was supported.

[yoojinl]

@thockin I will try to explain why I think it should be a part of Kubernetes. According to the quote from ConfigMap documentation:

Many applications require configuration via some combination of config files, command line arguments, and environment variables. These configuration artifacts should be decoupled from image content in order to keep containerized applications portable. The ConfigMap API resource provides mechanisms to inject containers with configuration data while keeping containers agnostic of Kubernetes.

The purpose of ConfigMap is to provide a way to decouple configuration from the container. Majority of the containers are configured using configuration files, and there are many different formats of configuration files. ConfigMap already tries to address this problem but only partly, because user cannot specify arbitrary configuration format.

Configuration of services includes not only storing the values, but also rolling out of the configuration, ConfigMap already solves it for initial configuration. Let’s consider a case when configuration is changed, Kubernetes owns update procedure, currently it is done for image changes, but it also should be seamlessly integrated with configuration changes procedure.

I agree with you that there are many ways to solve the problems, but I don’t think that there are that many straightforward, and error-prone ways to solve that from user's perspective.

Let me try to describe in more details the solution you suggest:

1. Use some templating system or confd to render configuration files in init containers.
2. At the same time I should have running instance of confd to monitor changes on a single node (or on multiple nodes + some locking mechanism) to run a chain of actions to restart containers. This single instance should have knowledge on what config is related to which service, so it can perform rollout procedure for subset of Pods.

Although technically it is definitely possible to make it work, it seems to be a bit grey area. Support of initial configuration and applications reconfiguration is definitely an important issue worth to be considered since every user has to deal with that and it is directly related to the purpose of Kubernetes which is automation of applications deployment.

[kfox1111]

@thockin I tried the init container for templating. It did work, but did run into an unusual issue.

Fails:
cp -a /srv/a/* /srv/b/; crudini --set /srv/b/foo.ini a b c;

Works
cp -a /srv/a/..data/* /srv/b/; crudini --set /srv/b/foo.ini a b c;

The config files at the root of the configmap are actually symlinks to stuff in the ..data dir. I'm not really sure why this is or if I should be relying on being able to read the data out of ..data/

Otherwise, the pattern worked quite well.

[thockin]

I didn't think you'd copy files, I expected to read files and render a
config from a template. Regardless, yes, the files are links. You'll need
to handle that

On Wed, Aug 17, 2016 at 2:55 PM, kfox1111 notifications@github.com wrote:

@thockin https://github.com/thockin I tried the init container for
templating. It did work, but did run into an unusual issue.

Fails:
cp -a /srv/a/* /srv/b/; crudini --set /srv/b/foo.ini a b c;

Works
cp -a /srv/a/..data/* /srv/b/; crudini --set /srv/b/foo.ini a b c;

The config files at the root of the configmap are actually symlinks to
stuff in the ..data dir. I'm not really sure why this is or if I should be
relying on being able to read the data out of ..data/

Otherwise, the pattern worked quite well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVN3hLIAPRv84lpzdoQRdqDRc2k80ks5qg4NjgaJpZM4JU_a8
.

[kfox1111]

Is it ok to edit them in place? Will anything overwrite them back to a pre rendered state?

[thockin]

Yes, they will be destroyed. They should be read-only to you in the first
place. If not, that's a bug :)

On Wed, Aug 17, 2016 at 5:46 PM, kfox1111 notifications@github.com wrote:

Is it ok to edit them in place? Will anything overwrite them back to a pre
rendered state?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVK6cGGeAWpjU8v-t4XtR3DGcJ5VRks5qg6tYgaJpZM4JU_a8
.

[kfox1111]

Ok, then coping them to an emptyDir is probably the correct behavior then.

What is the deal with the ..data/ dir though? will it always be the case that they are in ..data/? Are there any cases where it isnt?

[thockin]

..data gives us a way to do atomic updates of all the files at the same
time when the CanfigMap changes

On Thu, Aug 18, 2016 at 9:01 AM, kfox1111 notifications@github.com wrote:

Ok, then coping them to an emptyDir is probably the correct behavior then.

What is the deal with the ..data/ dir though? will it always be the case
that they are in ..data/? Are there any cases where it isnt?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVCkoFZ2bVRLAvs482ms9h5eTnAv4ks5qhIG8gaJpZM4JU_a8
.

[kfox1111]

I didn't think thats possible to do atomically? Near atomic I could see.
checkout to ..data2, mv ..data ..data.old; mv ..data2 ..data; rm -rf ..data.old

Is there a syscall to swap directories I don't know about?

Or is there yet another directory and ..data is a symlink? If that were the case, you could do a link swap atomically
ln -s ..data.newdir ..data.new; mv -Tf ..data.new ..data

[thockin]

There is a symlink involved :)

https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/atomic_writer.go

On Thu, Aug 18, 2016 at 10:15 AM, kfox1111 notifications@github.com wrote:

I didn't think thats possible to do atomically? Near atomic I could see.
checkout to ..data2, mv ..data ..data.old; mv ..data2 ..data; rm -rf
..data.old

Is there a syscall to swap directories I don't know about?

Or is there yet another directory and ..data is a symlink? If that were
the case, you could do a link swap atomically
ln -s ..data.newdir ..data.new; mv -Tf ..data.new ..data

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVD71jYghdQIvxF6dJ5KwC4Xiycy_ks5qhJNMgaJpZM4JU_a8
.

[kfox1111]

Ah. there it is. :)

Thanks. That clears it up in my head. :)

[thockin]

Sharp eyes :) The design is a little kludgey but it works pretty well in
practice.

On Thu, Aug 18, 2016 at 10:36 AM, kfox1111 notifications@github.com wrote:

Ah. there it is. :)

Thanks. That clears it up in my head. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29607 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVJK6fOE2Pjar5RdX43ZmjVbMzSmDks5qhJgCgaJpZM4JU_a8
.

[bgrant0607]

ConfigMap should remain just data.

This is effectively a dupe of #2068

See also #831