Add an inline volume type that stores small amounts of RO data with the manifest

[kamalmarhubi]

For cases where a small amount of static data is desired. This could include

1. configuration for containers whose process is configured by files
2. small-medium amounts of static data, such as an archive of www data for a static site server
3. inputs to a process that has a small input to work ratio

Example use cases:

• allow developers to specify data along side the pod it's used with, instead of needing to put it "somewhere else" (EBS / PD / RBD)
• allow Kubernetes-native process coordinators to pass input data to worker pods

For developers, this would be facilitated by a kubectl plugin (#13606) in a similar way to what is proposed for easing secret creation (#4822). The plugin would handle archive creation and encoding for inclusion in the manifest.

The size of this data would be limited, as it would be living in etcd. I don't know what a reasonable limit is as I'm not familiar enough with etcd. I would assume a couple hundred KB at the most?

[thockin]

Can you sketch the API in a bit more detail. I don't have any
philosophical objections to this.

On Sat, Sep 5, 2015 at 7:04 AM, Kamal Marhubi notifications@github.com
wrote:

For cases where a small amount of static data is desired. This could
include

1. configuration for containers whose process is configured by files
2. small-medium amounts of static data, such as an archive of www data for
   a static site server
3. inputs to a process that has a small input to work ratio

Example use cases:

• allow developers to specify data along side the pod it's used with,
  instead of needing to put it "somewhere else" (EBS / PD / RBD)
• allow Kubernetes-native process coordinators to pass input data to
  worker pods

For developers, this would be facilitated by a kubectl plugin (#13606
#13606) in a similar way
to what is proposed for easing secret creation (#4822
#4822). The plugin would
handle archive creation and encoding for inclusion in the manifest.

The size of this data would be limited, as it would be living in etcd. I
don't know what a reasonable limit is as I'm not familiar enough with etcd.
I would assume a couple hundred KB at the most?

—
Reply to this email directly or view it on GitHub
#13610.

[erictune]

Yes, I want this. It would be like this:

apiVersion: v1
kind: Pod
metadata:
  name: mypythonprogram
spec:
  containers:
   - name: c
      image: python
      command: ["python",  "/immediate/hello_server.py"]
    volumeMounts:
    - name: immediate
      mountPath: /immediate
volumes:
  - name: immediate
    immediateFiles:
      - name: hello.py
      - contents: |
#!/usr/bin/python
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
class myHandler(BaseHTTPRequestHandler):    
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type','text/html')
        self.end_headers()
        self.wfile.write("Hello World !")
        return
    server = HTTPServer(('', 8080), myHandler)  
    server.serve_forever()

[bgrant0607-nocc]

Do we really need this if we have ConfigData #1553?

[bgrant0607-nocc]

To clarify: Imagine something like Secret, but for non-secret data.

[kamalmarhubi]

@thockin sorry I've been traveling and only got to look at this now. Something like @erictune's example is what I had in mind, though with the ability to specify it as base64 encoded compressed data or something else like that.

@bgrant0607-nocc I think that would work for my imagined use cases. The ConfigData could be specified in the same file for easy reference. Similarity with Secret would be good for sharing client side stuff I remember reading about for creating the secrets from files &c.

[erictune]

I guess with ConfigData, it would look something like this:

apiVersion: v1
kind: Pod
metadata:
  name: mypythonprogram
spec:
  containers:
   - name: c
      image: python
      command: ["python",  "/immediate/hello_server.py"]
    volumeMounts:
    - name: immediate
      mountPath: /immediate
volumes:
  - name: file
    configData:
      configDataName: mypythonfile
----
apiVersion: v1
kind: ConfigData
metadata:
  name: mypythonfile
spec:
    files:
      - name: hello.py
      - contents: |
#!/usr/bin/python
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
class myHandler(BaseHTTPRequestHandler):    
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type','text/html')
        self.end_headers()
        self.wfile.write("Hello World !")
        return
    server = HTTPServer(('', 8080), myHandler)  
    server.serve_forever()

A little wordier, but works too.

[kamalmarhubi]

I did think of one reason to still want an inline volume: it would tie the lifetime of the volume to the pod. Imagine a Kubernetes aware application that creates child pods to do work. It may want to pass them different configuration, such as the shard of data to work on. Creating a ConfigData as well as the pod would require more cleanup, and could result in garbage ConfigData in case of errors.

I'm totally willing to accept YAGNI on this though! My use case is quite hypothetical at this point!

[erictune]

ConfigData is some other object that has to be named, created, deleted. It need to have its own permissions. This does not have that overhead. Which is nice.

Also, immediate data is immutable, and in some cases, you want to express that the configuration should
not change after runtime.

[bgrant0607]

On lifetime: we plan to support cascading deletion, so lifetime shouldn't be an issue. Just use ConfigMap.

[jberkus]

this would be excellent, and a nice alternative to templates.

One request: allow for input from a file on the host as well as inline contents.

[thockin]

You can effectively do this with a ConfigMap and it's even better -
ConfigMap is fully update-friendly, so the files can "change" without the
Pod restarting, if you want.

On Fri, Sep 2, 2016 at 3:18 PM, Josh Berkus notifications@github.com
wrote:

this would be excellent, and a nice alternative to templates.

One request: allow for input from a file on the host as well as inline
contents.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVC49TFjjH1EFSItOWX32ZDQ-fBm0ks5qmKDCgaJpZM4F4bqf
.

[derekmahar]

@bgrant0607

Given that ConfigMaps cannot (yet) contain binary data (see #32432), I think the idea of a pre-populated, read-only volume type is still relevant. For example, in order to deploy a binary file to a pod using a volume, I am torn between using a gitRepo or a ConfigMap volume. Each has benefits and drawbacks:

gitRepo

Advantages

• Can store common file types including text, binary, or directory.

Disadvantages

• Requires a remote Git repository which further requires an HTTP or HTTPS proxy and in the case of HTTPS, TLS certificates.
• Pull model that requires explicit volume update or a pod restart.

ConfigMap

Advantages

• Push model that automatically updates mounted volumes.

Disadvantages

• Cannot store binary files unless files are stored using base64 encoding.
• Cannot store directories.

Ideally, I'd like a volume type that combines the advantages of both gitRepo and ConfigMap, but without their disadvantages. This volume type would effectively enable me to push a populated file tree into one or more pods. I think this may have been an intention of #831, but I think that issue implies a Docker image, not just a simple file system image. In any case, I think the basic concept is still useful and not entirely addressed by ConfigMap and gitRepo volume types.

[thockin]

On Mon, Sep 19, 2016 at 11:50 AM, Derek Mahar notifications@github.com wrote:

@bgrant0607 @kamalmarhubi @erictune @thockin

Given that ConfigMaps cannot (yet) contain binary data (see #32432), I think the idea of a pre-populated, read-only volume type is still relevant. For example, in order to deploy a binary file to a pod using a volume, I am torn between using a gitRepos or a ConfigMap volume. Each has benefits and drawbacks:

gitRepos

Advantages

Can store common file types including text, binary, or directory.

Disadvantages

Requires a remote Git repository which further requires an HTTP or HTTPS proxy and in the case of HTTPS, TLS certificates.
Pull model that requires explicit volume update or a pod restart.

git-sync as a sidecar is a better answer than gitRepo volumes, and self-updates

ConfigMap

Advantages

Push model that automatically updates mounted volumes.

Disadvantages

Cannot store binary files unless files are stored using base64 encoding.
Cannot store directories.

ConfigMap can be expanded into directories, as long as you have a key per-file.

Binary data is true, but if Secret adds plain-text support, ConfigMap
could add Binary... maybe. @pmorie @smarterclayton

Secret can do Binary data...

Ideally, I'd like a volume type that combines the advantages of both gitRepos and ConfigMap, but without their disadvantages. This volume type would effectively enable me to push a populated file tree into one or more pods. I think this may have been an intention of #831, but I think that issue implies a Docker image, not just a simple file system image.

inline-volumes would have to be explicitly allowed to update - we do
not allow updates to most pod fields...

[derekmahar]

@thockin

git-sync as a sidecar is a better answer than gitRepo volumes, and self-updates

I read about git-sync on slide 16 of Brendan Burns' Composite Containers for Modular Architectures and @philips referred to it in a recent reply to How to perform automatic data copy/reset with Kubernetes ?, but even that I find more complicated than simply providing a volume that already contains some files.

ConfigMap can be expanded into directories, as long as you have a key per-file.

I'm glad you corrected me on this point, because I wasn't sure. It's still not clear to me exactly how to consume the directories in a ConfigMap volume, though.

Secret adds plain-text support, ConfigMap could add Binary... maybe. Secret can do Binary data...

I didn't know that Secret could store binary, but not plain text. I may use this as a workaround to store my binary files. It's a little awkward to split text and binary files between ConfigMap and Secret, but I prefer this option to creating a side car or using an init container. (I think init containers aren't yet available, anyway.) These are useful patterns, but in this case, I'm looking for something as easy to use as a ConfigMap.

inline-volumes would have to be explicitly allowed to update - we do not allow updates to most pod fields..

Before you replied, I was going to comment that #24044 more closely resembles what I have in mind than inline-volumes. @bgrant0607 closed #24044 as a duplicate of #831, so I guess a container volume type is even closer to what I want, but without the Docker image, if that is what #831 implies.

Would a pre-populated volume type be difficult to implement? I'm puzzled why #831 or similar isn't a higher priority. In fact, I'm quite surprised that it hasn't already been implemented. I think it's a really good idea and very easy to understand and would likely be just as easy to use.

[smarterclayton]

I think we need to fix the ConfigMap thing but it has a work around (use a secret).

Do we have to allow inline volumes to be updated? I think the value of an inline volume is more if they can't be updated (otherwise, use a config map). Treating an inline volume like a piece of static, materialized config co-located with the template that stamps them out (RS / PS / D) that can be safely changed in a rollout is (to me) the primary value they have for anything more complex than an env var and less complex than a full config-driven workflow.

#831 is very interesting especially with very small images, but has enough headwinds in how Docker handles those types of containers and the other changes in CRI that it's fairly risky today.

[thockin]

Secret can store both, but it base64 encodes everything. Eventually that
will be fixed. But that is a hack.

ConfigMaps can be explicitly mapped key-to-file in the volumes block, and
that can be paths like "foo/bar/bat.txt"

#831 is specifically about pulling container images, so "but without the
Docker image" is something else :) Where do you envision the data coming
from?

@smarterclayton: "Do we have to allow inline volumes to be updated" -
up-thread this was proposed as a requirement. Without that requirement it
is vastly easier.

On Mon, Sep 19, 2016 at 3:14 PM, Derek Mahar notifications@github.com
wrote:

@thockin https://github.com/thockin

git-sync as a sidecar is a better answer than gitRepo volumes, and
self-updates

I read about git-sync on slide 16 of Brendan Burns' Composite Containers
for Modular Architectures
http://www.slideshare.net/Docker/slideshare-burns and @philips
https://github.com/philips referred to it in a recent reply to How to
perform automatic data copy/reset with Kubernetes ?
https://groups.google.com/d/topic/kubernetes-users/ob437DP0tpY/discussion,
but even that I find more complicated than simply providing a volume that
already contains some files.

ConfigMap can be expanded into directories, as long as you have a key
per-file.

I'm glad you corrected me on this point, because I wasn't sure. It's still
not clear to me exactly how to consume the directories in a ConfigMap
volume, though.

Secret adds plain-text support, ConfigMap could add Binary... maybe.
Secret can do Binary data...

I didn't know that Secret could store binary, but not plain text. I may
use this as a workaround to store my binary files. It's a little awkward to
partition text and binary files between ConfigMap and Secret, but I prefer
this option to creating a side car or using an init container. These are
useful patterns, but in this case, I'm looking for something as easy to use
as a ConfigMap.

inline-volumes would have to be explicitly allowed to update - we do not
allow updates to most pod fields..

Before you replied, I was going to comment that #24044
#24044 more closely
resembles what I have in mind than inline-volumes. @bgrant0607
https://github.com/bgrant0607 closed #24044
#24044 as a duplicate of
#831 #831, so I guess a
container volume type is even closer to what I want, but without the Docker
image, if that is what #831
#831 implies.

Would a pre-populated volume type be difficult to implement? I'm having
some difficulty understanding why #831
#831 or similar isn't a
higher priority. In fact, I'm quite surprised that it hasn't already been
implemented. I think it's a really good idea and very easy to understand
and would likely be just as easy to use.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGBmhxhKhuIwhhZSqcFM-CGOYUCZks5qrwk6gaJpZM4F4bqf
.

[derekmahar]

@thockin

#831 is specifically about pulling container images, so "but without the Docker image" is something else :) Where do you envision the data coming from?

I imagined it would work similar to how kubectl creates a ConfigMap (kubectl create configmap configmap-name --from-file=directory) where a directory on the kubectl host populates an object in the cluster that would back the volume. A volume would associate this directory with a volume name which a container could then map into its file system. I think this is similar to what you described for a ConfigMap directory. The idea is very similar to the directory tree in a ConfigMap, but but without the key-value mapping. I guess I could just use a ConfigMap this way, once it supports binary files.

[smarterclayton]

I think I would argue that updates belongs to configmaps and secrets (because you have to deal with them), immutable things belong to pods (because they are easier to reason about). The initial issue talks about immutable pod data (where config + pod definition = total config).

Discussing the complexity of achieving safe config map reuse in deployments convinced me that mutable config maps are more dangerous than they should be - I think we should look for affordances that reduce the complexity of mutable config maps and also make simple pods easier to write. I generally agree that inline volumes are rife for abuse - but if they are immutable, they do focus the attention on a workflow we support for other pod level attributes in a very user-oriented way.

An argument along this line is that env variables can be very large - promoting someone using an env var to parameterize a file by using a bash entrypoint isn't wrong, but it is complex, and is the user workaround the spirit rather than intent. It is a totally valid workaround once we fix the downward API to allow more fields to do:

env:
- name: CONFIG
   value: |- 
     massive chunk of config
volumes:
  downwardAPI:
    valueFrom:
    - fieldPath: spec.containers[1].env[@name=CONFIG].value

I'm just not sure we should encourage it.

[thockin]

I would be OK with inline volumes, if someone wants to craft the smallest
possible proposal

On Mon, Sep 19, 2016 at 4:52 PM, Clayton Coleman notifications@github.com
wrote:

I think I would argue that updates belongs to configmaps and secrets
(because you have to deal with them), immutable things belong to pods
(because they are easier to reason about). The initial issue talks about
immutable pod data (where config + pod definition = total config).

Discussing the complexity of achieving safe config map reuse in
deployments convinced me that mutable config maps are more dangerous than
they should be - I think we should look for affordances that reduce the
complexity of mutable config maps and also make simple pods easier to
write. I generally agree that inline volumes are rife for abuse - but if
they are immutable, they do focus the attention.

An argument along this line is that env variables can be very large -
promoting someone using an env var to parameterize a file by using a bash
entrypoint isn't wrong, but it is complex, and is the user workaround
the spirit rather than intent. It is a totally valid workaround once we fix
the downward API to allow more fields to do:

env:

• name: CONFIG
  value: |-
  massive chunk of config
  volumes:
  downwardAPI:
  valueFrom:
  • fieldPath: spec.containers[1].env[@name=CONFIG].value

I'm just not sure we should encourage it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGv1QuWPXg5biT9JR-R3TXN_Z5OYks5qryBGgaJpZM4F4bqf
.

[thockin]

But where would we store that data in the API as it transits from your
computer to the API server and down to a kube node? An inline volume or
configmap would be good for this.

On Mon, Sep 19, 2016 at 3:36 PM, Derek Mahar notifications@github.com
wrote:

@thockin https://github.com/thockin

#831 #831 is
specifically about pulling container images, so "but without the Docker
image" is something else :) Where do you envision the data coming from?

I imagined it would work similar to how kubectl creates a ConfigMap where
a directory on the kubectl host populates the object or volume type that
would back the volume. A volume would associate this directory with a
volume name which a container could then map into its file system. I think
this is similar to what you described for a ConfigMap directory.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVKylv9nYC_3_kI0mPdkZRaGJqsfeks5qrw5vgaJpZM4F4bqf
.

[smarterclayton]

Doh - I just realized we have annotations for downward API (plan to). So we don't need inline volume if we can figure out how to handle binary data (downward API allows base64 decoding).

[smarterclayton]

If we add that flag, that solves config map as binary as well. Then create can support that scenario by passing an additional flag.

[smarterclayton]

Simple proposal:

1. Leave configmaps as they are today
2. Support annotation projection from downward API and env var the same
3. Add an "decodeBase64 bool" to the downward API volume and env var projections
4. Support "base64encode" on kubectl create
5. Give example for annotation in base64 for inline config
6. Profit

[thockin]

I'd almost rather have the volume - its conceptually much cleaner...

On Mon, Sep 19, 2016 at 5:24 PM, Clayton Coleman notifications@github.com
wrote:

Simple proposal:

1. Leave configmaps as they are today
2. Support annotation projection from downward API and env var the same
3. Add an "decodeBase64 bool" to the downward API volume and env var
   projections
4. Support "base64encode" on kubectl create
5. Give example for annotation in base64 for inline config
6. Profit

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVDBnEh7Aiir6vcdkFlk5yd4mbeAiks5qryfUgaJpZM4F4bqf
.

[smarterclayton]

I can think of several cases for env vars where I need json not to eat my values.

So follow up questions:

1. Should I be allowed to set binary values in annotations?
2. Should I be allowed to use those binary values in downward API?
3. Should the CLI support me setting those values?

I do think if annotations can solve the non binary case, then the binary case is really just an edge case of that. And annotations can be set by a wide range of tools - do we need two ways to do the same thing (store app specific metadata on a pod definition)?

[thockin]

How do you denote that an annotation is a binary value?

On Mon, Sep 19, 2016 at 7:23 PM, Clayton Coleman notifications@github.com
wrote:

I can think of several cases for env vars where I need json not to eat my
values.

So follow up questions:

1. Should I be allowed to set binary values in annotations?
2. Should I be allowed to use those binary values in downward API?
3. Should the CLI support me setting those values?

I do think if annotations can solve the non binary case, then the binary
case is really just an edge case of that. And annotations can be set by a
wide range of tools - do we need two ways to do the same thing (store app
specific metadata on a pod definition)?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVKBMKbTB7D-FFoPpOayPxSNvi_cvks5qr0OXgaJpZM4F4bqf
.

[smarterclayton]

At the point of projection in downward API - "use this value and treat it as base64"

[thockin]

I guess I don't have a huge problem with env->projection it's just kind of
weird to me. I still think an inline would be more obvious and discoverable

On Tue, Sep 20, 2016 at 6:38 PM, Clayton Coleman notifications@github.com
wrote:

At the point of projection in downward API - "use this value and treat it
as base64"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVPZrHkA8TTqKlBm_6dE8ZaRkEclSks5qsIqPgaJpZM4F4bqf
.

[smarterclayton]

You feel annotation projection is weird too? One nice thing of making annotation capable of direct injection for both binary and normal text is that it is a single example to show and explain.

[thockin]

Annotation feels more obvious. I just worry about the cross-links. Before
long it will be possible to project annotations as files or and env, and to
project env as file. If projecting annotations was allowed to file and
env, then we could perhaps de-dup the logic and have a consistent model.

On Wed, Sep 21, 2016 at 6:28 AM, Clayton Coleman notifications@github.com
wrote:

You feel annotation projection is weird too?

One nice thing of making annotation capable of direct injection for both
binary and normal text is that it is a single example to show and explain.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVBFBi8IXy10Js4LgSq6ZWFNPKAIeks5qsTEWgaJpZM4F4bqf
.

[smarterclayton]

I think we plan to de-dup the logic - @pmorie and I plotted it, but did not
execute in 1.4 because of other situations. I am assuming a world where
env and volume do exactly the same thing.

On Wed, Sep 21, 2016 at 11:09 AM, Tim Hockin notifications@github.com
wrote:

Annotation feels more obvious. I just worry about the cross-links. Before
long it will be possible to project annotations as files or and env, and to
project env as file. If projecting annotations was allowed to file and
env, then we could perhaps de-dup the logic and have a consistent model.

On Wed, Sep 21, 2016 at 6:28 AM, Clayton Coleman <notifications@github.com

wrote:

You feel annotation projection is weird too?

One nice thing of making annotation capable of direct injection for both
binary and normal text is that it is a single example to show and
explain.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/kubernetes/kubernetes/issues/
13610#issuecomment-248612012>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AFVgVBFBi8IXy10Js4LgSq6ZWFNPKAIeks5qsTEWgaJpZM4F4bqf>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13610 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p9f5mes391ddw7fReuhgYhR5TkjXks5qsUi_gaJpZM4F4bqf
.

[h0jeZvgoxFepBQ2C]

So is there a solution like here? #13610 (comment)

[thockin]

ConfigMap is the answer we have