Integrate third-party configuration management tools with our config distribution API

[stp-ip]

This issue is based on the suggestions in #1553. I thought getting a clean overview for a renewed discussion was worth the additional issue.

Similar to secret distribution #2030 there is a need to dynamicaly configure applications. Producing or changing configuration scripts via cat, sed or other "hacky" tooling is a method I dislike and I would love a cleaner way of supporting this.

Option 1: envconsul/etcdenv

Pros:

• Injects etcd/consul values as ENVs
• Easy way to switch from ENVs to etcd/consul

Cons:

• Doesn't actually solve config generation

Option 2: confd

Pros:

• Supports various backends (consul, etcd, ENVs)
• Supports almost any config structure as it is template based
• Easy templates using Golang Text Templates
• Support for one-time and watch (etcd/consul)
• Command line flags to bootstrap confd config

Cons:

• Template based
• No encrypted value support -> crypt or just encrypted volumes Secret distribution in docker/k8s #2030?

Option 3: viper

Pros:

• Supports encrypted values via crypt
• No templates
• Easy syntax
• Command line flags to bootstrap viper config

Cons:

• No ini file support
• Not as easy for complex configuration files

Option 4: augeas

Pros:

• Wide range of config formats supported

Cons:

• Complex syntax especially for complex configurations

Option5: tiller

Pros:

• Supports almost any config structure as it is template based
• Ability to "pipe" json configs into a container instead of ENV files

Cons:

• Template based
• ERB templates -> more complex than necessary in my opinion
• Can use container provided .yaml files -> could prevent image sharing

Clarification:

Confd:

@kelseyhightower
Is a config file needed for confd to work or could everything be done using commandline flags?
Can multiple config files be generated, if yes how?

Viper:

@spf13 @bketelsen
Is a config file needed for viper to work or could everything be done using commandline flags?
-> Config file and various other configuration possibilities for viper are optional. It can be used straight via commandline flags.

Overall I think the way to go would be to use confd with ENV as default backend. Furthermore containers should be able to understand a backend switch ENV so that switching to etcd/consul is easily done on a per container level at runtime. This enables easy testing and environment switching. With the addition of watchable configuration via confd more dynamic setups are possible.

Looking forward to your thoughts and suggestions.

[stp-ip]

cc @jbeda @kelseyhightower @bketelsen @spf13 @thockin @bgrant0607

[spf13]

Viper works completely without a config file. Viper can use defaults, confd, environmental, config and flags if available. All are optional. =

[stp-ip]

@spf13 thanks for the clarification. Will update accordingly. If you have any other thoughts, feel free to add them.

[jbeda]

I love that we are thinking about this type of stuff. From a Kubernetes point of view:

• Is this something that should be built into k8s or is it layered on top?
• Are there generic-ish features that k8s could add that would enable these things to work a lot smoother?
• Is it worth perhaps handling this in documentation and examples as a common pattern/tool used with k8s?

[stp-ip]

@jbeda
Builting into k8s is probably hard as config stuff would need to run inside the container and that's dockers area. I'm looking into suggesting something like confd in docker itself with the new dockerfile2 etc. -> this would dismiss a layer on top too.
The only way I could imagine k8s could improve it from within k8s, would be a way to mount a config volume. This volume could be launched via k8s with a seed secret, templates and etcd access and would generate secrets/configs and make these available to the launched container. It would make things smoother similar to the git backed volume, but it would reduce modularity and reusability I think. So more input is needed.

I think the first generic feature would be to have etcd/consul backend for k8s and make both be usable from containers. So that a second cluster for both wouldn't be needed. Additionally ACL is needed to prevent config leakage and security issues, when a container has access to the etcd/consul cluster. -> ACL is more an issue with etcd/consul themselves most likely.

I think the main need is better documentation and best practices on top of k8s for now. So that people arriving at k8s can use these guidelines to either build a usual docker cluster or jump right on k8s.

For now we should try to find a common ground on how to do things like config, secret and general best practices so that we can improve the documentation, provide a guideline/format for docker containers and later on jump into integrating these more closely into k8s. As mentioned in #2030 mainly key rotation and seed secret injection from k8s.

[stp-ip]

An additional thought, which came to mind:
With the support from openshift perhaps and the mentioned best practices we could push for "fully" reusable and modular images. The current available images even the official ones lack most of the customization, configuration and secret needs we are discussing here. This could lead to (don't worry quite far off speculation) a quick start for new users with quite proofen base images and reduce duplication of work in my opinion.

[bketelsen]

@stp-ip you dragged me over to an interesting idea: When you suggested having etcd/consul backend for k8s it occurred to me that it would be pretty simple to write a proxy for one of these two stores and service requests coming from pods while transparently encrypting/decrypting the values. Unencrypted data would never leave the pod's network, in theory never leaving the host at all. Just throw that in the pile of ways that we could solve these problems. It adds some fragility because the backing stores will most certainly evolve over time, causing code maintenance.

I realize that there are a million ways to solve this problem, and @jbeda rightly suggests that the problem domain might not necessarily belong to k8s. I think I could argue both sides of that pretty effectively. But secrets and keys are a big deal, and there should certainly be good documentation of strong patterns if it ends up being a problem left to be solved outside of k8s proper. When left to our own devices, we programmers don't always make the most secure choices.

[rektide]

I'd like to provide some additional links that might not directly contribute to this discussion,

I've filed a ticket for support for dynamic configuration at the systemd level via an ExecutionOneshot= parameter.

There's also Stocker which is a very fancy Docker-targeting repository for Environment config. I don't know enough about it and where it would fit in with this ticket to know whether it should be an option up top, but perhaps others can better evaluate this candidate option.

[stp-ip]

@rektide already reading up on stocker and will evaluate how to add it to #2030 and #2068. Will update the original post accordingly.

[thockin]

I love the idea of a confdir volume type. The trick is that we either pick
a winner, support everything, or provide a loose enough abstraction.

All three have their downsides.

On Fri, Oct 31, 2014 at 11:49 AM, Michael Grosser notifications@github.com
wrote:

@rektide https://github.com/rektide already reading up on stocker and
will evaluate how to add it to #2030
#2030 and #2068
#2068. Will
update the original post accordingly.

Reply to this email directly or view it on GitHub
#2068 (comment)
.

[stp-ip]

Going along with my proposal I want to suggest a "solution":

As we won't be able enable every tool to work with a given application/environment and additionally the preferences for tools are quite different, I'm proposing the usage of a more generic abstracting solution.
This solution would make a few things possible:

• Run images in a default mode, without any configuration
• Use various tools to define/generate configurations
• Watch on configuration changes
• Enable image sharing without defining the actual config generation
• Provide a modular way to decouple base image and config
• Make configuration development easier with manual config support

My proposed solution is to go the route of using Data Volume Containers or the equivalent. Therefore each docker base image is relying on default configuration available inside the container in /custom/configuration/. If no volumes are mounted on /custom/, then the default files in /custom/* are used.
Basic image testing can therefore be done easily. More advanced runtimes can then easily link the volume to the host (development setup), a git volume #1945 or the preferred way to add it via a Data Volume Container.
Furthermore this container can then use a base config image of choice (etcd, consul, confd, etcdenv etc.) and layer the actual application specifics on top (templates, filenames, locations etc.).

Additionally I imagine using the filesystem as a way to encourage config updates and guidelines.
UPDATE:
A method (not sure about the approach yet) should scan for an UPDATE file in the /custom/configuration/ directory and restart/reload the base image to reflect the changes.
Therefore all config changes without any distinction of the tools used can trigger an process reload, which is defined in the actual app base image. So the app vendor can control the restart policy etc.

The Structure for the filesystem:
As mount directory I suggest to use /custom/, with a view subdirectories:

• /configuration: mount point for configuration files -> generated via a base config image
• /secret(s): mount point for de/encrypted secrets (not yet defined on how to store secrets Secret distribution in docker/k8s #2030)
• /data: mount point for "persistent" data (databases, uploads etc.)
  If nothing is mounted default data is present.

In the future k8s could provide an integrated way to generate such volumes similar to providing git based volumes.

Any objections/suggestions for this best practice idea with a possible future agenda of integration into k8s?

[stp-ip]

As a sidenote: Is specifying a tag for a Data Volume Container possible? So that I can say mount project1/nginx-conf:2014-10-10 in the pod definition?

[stp-ip]

Any ideas on how to listen to updates via an UPDATE file? New config run touches UPDATE, file update is noticed by container (how?), UPDATE is removed.

[kelseyhightower]

@stp-ip confd does not require a config file, but does require templates and template configs be stored under a configdir, but the configdir and be specified via the command line.

[kelseyhightower]

When it comes to configuration for containers I prefer to pull configs out of etcd or just use a configuration volume.

[stp-ip]

@kelseyhightower Thanks for the clarification on confd.

With my recent suggestion at #2068 (comment) I think we got a generic way of accomplishing both. The usage of a preferred tool (confd for example) and configuration volumes. It makes images much more modular and interchangeable and introduces separation between the configuration container and the service container.

[bketelsen]

this is a solid suggestion that covers most of the use cases I can imagine

[hydrajump]

@kelseyhightower do you use a "data-only" container to hold the configuration per container, so for instance an nginx container will have its own nginx-config container?

Why not COPY the config into the image?

I prefer to pull configs out of etcd

Are you grabbing a complete config file, e.g. /etc/nginx/conf.d/default.conf or just environment variables?

[smarterclayton]

Re: convention vs configuration for mounted volumes, if the infrastructure provides the abstraction for the volume changing you don't necessarily need to define a convention (pod authors can specify the necessary binding). The real trick is in Docker decorating the volume definition of the image sufficiently to allow easier automating of that binding.

How do things like config volumes play with pods and their templates? When I want to create the pod and parameterized the source of a config volume, how easy is that for the pod consumer?

[smarterclayton]

@bketelsen allowing easy proxying / virtualization of an etcd key space for a particular application has a lot of advantages the larger your deployment grows - such as scaling the infrastructure cost of running one etcd per app topology into one etcd cluster to N consumers. Kelsey had provided me some examples and it's something we've considered exposing. Being able to encrypt at rest would be ideal (esp if the proxy was a component of your namespace, not part of shared infra)

[stp-ip]

This is the basic structure I imagined taking into account #2030:

If anything is unclear, I'm around.
Nice thing about volume usage is that, when standardised, switching the "/custom/configuration"-container to use etcd instead of consul or using a manual config file in development etc. is easily done. Furthermore it's easy to provide volumes as a service for config, data and secrets (+ providing seed key injection via /custom/key).

[stp-ip]

On a sidenote:
How is GCE doing persistent storage for containers?

[erictune]

An alternative proposal to distribute secrets to a trusted proxy instead of the pods themselves.
#2209

Won't cover all the cases, but might cover a lot of them.

[bgrant0607]

I like the idea of decoupling configuration generation from the distribution and consumption mechanisms. We want to do that regardless what the latter 2 mechanisms look like.

Data volume containers were proposed in #831. That seems useful regardless whether we use it for this.

@smarterclayton Had a good point of how configuration volumes would affect Pod and PodTemplate specifications.

Specifying the configuration image version explicitly in pods improves predictability and transparency, but makes configuration deployment about as heavy-weight as deploying a whole new image. Command-line flags and environment variables passed via Docker have the same problems.

If we wanted a way to update configuration volumes without starting a container, that would require a new in-place update mechanism. Do we want the Pod and PodTemplate specifications and even bound pods (and Docker containers) to be updated in place? In-place updates improve application availability, but introduce a lot of complexity into the application management ecosystem.

One could update the volume from inside the container.

[pmorie]

@stp-ip Ack to your earlier comment, I am reviewing your linked docker proposal and a slew of other similar work, thanks.

[smarterclayton]

The umbrella issue for security right now is #4029, with the sub proposals linked.

On Feb 2, 2015, at 10:50 PM, Paul Morie notifications@github.com wrote:

@stp-ip Ack to your earlier comment, I am reviewing your linked docker proposal and a slew of other similar work, thanks.

—
Reply to this email directly or view it on GitHub.

[bgrant0607]

Retitled this issue to: Integrate third-party configuration management tools with our config distribution API.

I'm using #1553 for the configuration distribution API and basic tooling. #1627 is for using that for control-plane configuration.

[gurvindersingh]

/subscribe

and looking forward to dynamic configuration using etcd/consul

[gurvindersingh]

any update on this issue or plan to have it included in the 1.1 k8s release ?

[hwinkel]

is there any progress on this topic? I'm wondering what the final strategy would be. Managing this externally via salt, ansible, chef etc. or leverage the existing meta data stores like etcd or even consul.

[onbjerg]

Any progress or updates on this? The thread seems eerily quiet. Is it because there's not enough resources to allocate, or because a reasonable solution wasn't found?

Because as I see it, some of the bigger workflow improvements for k8s would be this issue and #23896.

[erictune]

ConfigMaps are what we recommend. Documented here: See here: http://kubernetes.io/docs/user-guide/configmap/

Also, they can be dynamically updated. This fact isn't documented well. See:
https://github.com/kubernetes/kubernetes/blob/master/docs/design/configmap.md#consuming-configmap--updates

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

This is obsolete