Projecting single files from volumes into a container's FS #19764

pmorie · 2016-01-16T19:37:03Z

This issue is to explore the possibility of allowing users to make projections of individual files from volumes into a container. Example use-case is where the user wants to place volume content into an existing directory without mounting over the existing contents of that directory. Example: say you want to project a file into /etc/ssl/certs without making the other files in that directory inaccessible.

@mfojtik This is your use-case -- feel free to elaborate.

@thockin @smarterclayton @bgrant0607 @saad-ali @kubernetes/rh-storage

The text was updated successfully, but these errors were encountered:

pmorie · 2016-01-17T01:31:31Z

After some thought, I realize that if we do this, there's really no need for #19762

thockin · 2016-01-17T05:44:35Z

So the biggest issue with this is that updating a file atomically across a
namespace is practically impossible. You have to enter the target
namespace to really do it right.

I'd love to see this be possible, so it is probably worth exploring, but I
think it HAS to support atomic updates properly if updates are supported at
all.

On Sat, Jan 16, 2016 at 11:37 AM, Paul Morie notifications@github.com
wrote:

This issue is to explore the possibility of allowing users to make
projections of individual files from volumes into a container. Example
use-case is where the user wants to place volume content into an existing
directory without mounting over the existing contents of that directory.
Example: say you want to project a file into /etc/ssl/certs without
making the other files in that directory inaccessible.

@mfojtik https://github.com/mfojtik This is your use-case -- feel free
to elaborate.

@thockin https://github.com/thockin @smarterclayton
https://github.com/smarterclayton @bgrant0607
https://github.com/bgrant0607 @saad-ali https://github.com/saad-ali
@kubernetes/rh-storage
https://github.com/orgs/kubernetes/teams/rh-storage

—
Reply to this email directly or view it on GitHub
#19764.

mfojtik · 2016-01-17T08:53:43Z

@pmorie thanks! another use-case we have is to add files like .npmrc into $HOME or to /etc. This file contains internal URL to nodejs mirror and credentials (user+pass). This file is needed during the build time and not needed in output image (so having a way to 'unplug' the file from the contains would be nice to have as well ;-)

mfojtik · 2016-01-17T08:55:04Z

@thockin I think the update to secret content should trigger restart of the container where the secret is projected to.

thockin · 2016-01-17T18:29:32Z

This is a point of debate for secrets. DownwardAPI already allows dynamic
refresh, and config map is not done yet.
On Jan 17, 2016 12:55 AM, "Michal Fojtik" notifications@github.com wrote:

@thockin https://github.com/thockin I think the update to secret
content should trigger restart of the container where the secret is
projected to.

—
Reply to this email directly or view it on GitHub
#19764 (comment)
.

bgrant0607 · 2016-01-21T19:24:30Z

Some applications expect configuration files directly in /etc. This would be useful for such cases.

Yes, we need atomic updates for some cases.

pmorie · 2016-01-21T20:52:31Z

I think this is also solvable with pre-start hooks

thockin · 2016-01-22T01:00:14Z

Apps that NEED it in /etc/foo.conf can provide an image that symlinks
/etc/foo.conf -> /vol/etc/foo.conf.

I'm not against this feature, I think it would be great, I just don't think
the implementation (at least the one I know how to do) is justified, when a
relatively easy workaround is present.

On Thu, Jan 21, 2016 at 12:53 PM, Paul Morie notifications@github.com
wrote:

I think this is also solvable with pre-start hooks

—
Reply to this email directly or view it on GitHub
#19764 (comment)
.

pmorie · 2016-01-22T16:46:25Z

@thockin Agree, if you really want this, you can have it without a kube feature.

@mfojtik How do you feel about setting up symlinks in the container entrypoint, does that get you any closer to where you want to be?

smarterclayton · 2016-01-22T18:09:28Z

Changing the image is actually a fairly involved process - the statement we
make in other places is that adaptation of existing images to run is useful
and justifies some additional complexity on our end to make easy.
Individual file projection is useful in some cases, and less useful in
others. As someone who has done the symlinking thing, it's error prone,
annoying to do, and you end up doing it in a lot of images (if you have
more than 2 you care about). So there's some value in having a bit more
complexity. I don't know that it's required right now

On Fri, Jan 22, 2016 at 11:46 AM, Paul Morie notifications@github.com
wrote:

@thockin https://github.com/thockin Agree, if you really want this, you
can have it without a kube feature.

@mfojtik https://github.com/mfojtik How do you feel about setting up
symlinks in the container entrypoint, does that get you any closer to where
you want to be?

—
Reply to this email directly or view it on GitHub
#19764 (comment)
.

thockin · 2016-01-22T20:48:08Z

I agree this would be a great feature, I don't think it is "a bit more
complexity", though. If someone wants to prototype it, I'm happy to brain
dump (or just go look at moby/moby#17190)

On Fri, Jan 22, 2016 at 10:10 AM, Clayton Coleman notifications@github.com
wrote:

Changing the image is actually a fairly involved process - the statement we
make in other places is that adaptation of existing images to run is useful
and justifies some additional complexity on our end to make easy.
Individual file projection is useful in some cases, and less useful in
others. As someone who has done the symlinking thing, it's error prone,
annoying to do, and you end up doing it in a lot of images (if you have
more than 2 you care about). So there's some value in having a bit more
complexity. I don't know that it's required right now

On Fri, Jan 22, 2016 at 11:46 AM, Paul Morie notifications@github.com
wrote:

@thockin https://github.com/thockin Agree, if you really want this,
you
can have it without a kube feature.

@mfojtik https://github.com/mfojtik How do you feel about setting up
symlinks in the container entrypoint, does that get you any closer to
where
you want to be?

—
Reply to this email directly or view it on GitHub
<
#19764 (comment)

.

—
Reply to this email directly or view it on GitHub
#19764 (comment)
.

erictune · 2016-02-12T19:23:32Z

A customer had a narrower problem which might have a more direct solution -- no sure if it is worthwhile.
The customer wanted to have secrets A and B, each with one key, say "file-a" and "file-b". The customer wanted to mount them at "/etc/secrets/file-a" and "/etc/secrets/file-b". I suppose we could recognize this specific case at kubelet and merge the keys into a single combined volume prior to mounting. But, there are probably enough sharp corners that I'm not sure this is worthwhile. Also, explaining what is and isn't supported would get harder.

thockin · 2016-02-12T19:56:55Z

Options I see:

a) do the really complicated process for updating a file-projection (enter
target namespace...)

b) document that single-file projections are not live-updatable

c) document the (user-driven) workaround of making an image with
/etc/secret-a be a symlink -> /mnt/secrets/a

d) unpack the volume image ourselves and reconstruct the parent dir and
mount the whole dir (likely to break in other ways, e.g. docker commit)

e) convince upstream kernels to support bind-mounting symlinks as symlinks
(without dereference) This is a good answer if we can take a long view
(though it may be a nonstarter, I don't know kernel VFS that well)

On Fri, Feb 12, 2016 at 11:24 AM, Eric Tune notifications@github.com
wrote:

A customer had a narrower problem which might have a more direct solution
-- no sure if it is worthwhile.
The customer wanted to have secrets A and B, each with one key, say
"file-a" and "file-b". The customer wanted to mount them at
"/etc/secrets/file-a" and "/etc/secrets/file-b". I suppose we could
recognize this specific case at kubelet and merge the keys into a single
combined volume prior to mounting. But, there are probably enough sharp
corners that I'm not sure this is worthwhile. Also, explaining what is and
isn't supported would get harder.

—
Reply to this email directly or view it on GitHub
#19764 (comment)
.

kelseyhightower · 2016-03-07T06:53:05Z

I'm running into an issue right now where I'm trying to reuse an existing nodejs application which insists on reading a config.js file from it's application root directory. I was really hoping I could use a secret or configmap to inject the single config.js file and avoid "hiding" the application directory, but that does not work.

So yeah, this would be a really nice feature.

thockin · 2016-03-07T14:38:40Z

It is not sufficient or workable to build the image with the file being a symlink to an empty dir, and let secret or configmap mount into the dir? Not that I don't want this feature, but I want to see the f we can unblock people.

eparis · 2016-03-07T15:15:53Z

I know that @derekwaynecarr also had this problem. openshift expects certs and config files to live next to each other in the same directory. Yes, his problem is 'solvable'/'work-arround-able' by creating a custom image with symlinks. But custom images come with their own pain when the original image is updated.

smarterclayton · 2016-03-08T01:07:03Z

It's fairly common to have secrets and config mixed (off hand I think most Java app servers, some webservers, and lots of normal software have the same directory). I suspect most people would just assume it would work (that things would end up into the same location). I suspect they'd be willing to accept one or the other "winning". It's not the end of the world, but I suspect we'll have lots of users using config map and secret hit this, then try to do an ugly workaround.

pmorie · 2016-03-08T01:17:53Z

@smarterclayton @eparis @thockin Why not have an all-in-one volume for this for the time being?

thockin · 2016-03-19T05:42:48Z

The all-in-one was really ugly, I thought. I guess I am not really against it, it just feels clumsier than it should be. any other answer I have come up with is uglier, though. Still doesn't solve the as-a-file case.

deitch · 2017-02-19T11:34:46Z

it does work, now, but no updates

Does it? I could not figure out how. If I have an app that expects a config file in /app.conf or /app/conf/app.conf, what syntax do I use to mount the configmap as the single file app.conf without mounting over the entire / or /app/conf/?

deitch · 2017-02-19T11:43:14Z

Well, it looks like I figured it out here dshulyak/kubernetes.github.io@d58ba7b

but that was a lot harder to find than it should have been.

bryanlarsen · 2017-03-02T19:42:36Z

getting @deitch's example into the ConfigMap docs would be nice.

bitmage · 2017-03-30T18:49:10Z

I'm using ployst/nginx-ssl-proxy which looks for all its secrets in /etc/secrets. This worked fine when I just had secrets coming from one place (the letsencrypt container), but now that I want to add basic auth, things get complicated. I first tried:

command: ["/bin/sh", "-c", "ln -s /etc/auth/htpasswd /etc/secrets/htpasswd && ./start.sh"

I wasn't able to mount the cert secrets to /etc/secrets and then just symlink the auth secret. The symlink command would run successfully, but when the container started the htpasswd file would be gone. If I then exec -it [pod] /bin/bash I can manually run the symlink command and the file persists. This makes me think that maybe a volume refresh is being triggered during the container finalization lifecycle. Speculation though - there's really no evidence of what happened.

I ended up mounting both of my secrets to separate locations, then symlinking all the files I need:

command: ["/bin/sh", "-c", "mkdir -p /etc/secrets && \
                            ln -s /etc/certs/dhparam /etc/secrets/dhparam && \
                            ln -s /etc/certs/proxycert /etc/secrets/proxycert && \
                            ln -s /etc/certs/proxykey /etc/secrets/proxykey && \
                            ln -s /etc/auth/htpasswd /etc/secrets/htpasswd && \
                            ./start.sh"

It's an unfortunate amount of jumping through hoops though.

deitch · 2017-03-30T18:53:04Z

It's an unfortunate amount of jumping through hoops though

Bit of a theme....

thockin · 2017-03-31T04:08:47Z

@bitmage if you can boil down a repro, we can look into it. Naively, I would expect what you described to work...

@jingxu97 @saad-ali

debianmaster · 2017-04-16T02:20:15Z

facing same issue, any officially suggested workarounds for this problem?

debianmaster · 2017-04-16T02:57:18Z

nm found it!

This volumemount setup just injects agent.conf, and avoids mounting the entire /var/lib/juju/agents/application-tag/ directory. This way we don't lose the other things that juju puts in there. This setup doesn't allow kubernetes to update agent.conf when the ConfigMap changes, however, because of the way the mount works. See kubernetes/kubernetes#19764 (comment) Signed-off-by: Michael McCracken <mike.mccracken@canonical.com>

fearphage · 2017-05-01T14:00:44Z

@debianmaster Care to share the official response that you found?

schmitzhermes · 2017-05-26T11:01:19Z

Just spend several minutes searching for a solution, found it here: #23748 (comment)

you need to supply the absolute path including the filename in the mountPath and the filename again in subPath. The filename (obviously) needs to match the key in your config map.

debianmaster · 2017-05-26T15:35:53Z

@schmitzhermes @fearphage sorry for not getting back. you have the right solution @schmitzhermes it wont work from cli though if you have more than 1 file to mount in the same directory. i.e. only works for 1 file

This volumemount setup just injects agent.conf, and avoids mounting the entire /var/lib/juju/agents/application-tag/ directory. This way we don't lose the other things that juju puts in there. This setup doesn't allow kubernetes to update agent.conf when the ConfigMap changes, however, because of the way the mount works. See kubernetes/kubernetes#19764 (comment) Signed-off-by: Michael McCracken <mike.mccracken@canonical.com>

ketchoop · 2017-07-02T12:37:40Z

@schmitzhermes unfortunately, as I can see, configmap can't be updated when subPath is defined :(

thockin · 2017-07-02T15:56:33Z

This is not specific to configmap but to any subpath projection.

…

On Jul 2, 2017 5:37 AM, "Boyko Alexey" ***@***.***> wrote: @schmitzhermes <https://github.com/schmitzhermes> unfortunately, as I can see, configmap can't be updated when subPath is defined :( — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#19764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVC5eqQSuicV6-HuDwr88aDsC2D6wks5sJ48jgaJpZM4HGVRn> .

smarterclayton · 2017-07-03T02:37:42Z

I cry a little bit every time I hit this limitation. It's definitely one of my top 10 usability issues for any complex pod (since a lot of Unix software uses `/etc/` and we can't project into `/etc` without a hammer or hacks. On Sun, Jul 2, 2017 at 11:56 AM, Tim Hockin <notifications@github.com> wrote:

…

This is not specific to configmap but to any subpath projection. On Jul 2, 2017 5:37 AM, "Boyko Alexey" ***@***.***> wrote: > @schmitzhermes <https://github.com/schmitzhermes> unfortunately, as I can > see, configmap can't be updated when subPath is defined :( > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/kubernetes/issues/ 19764#issuecomment-312489424>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFVgVC5eqQSuicV6- HuDwr88aDsC2D6wks5sJ48jgaJpZM4HGVRn> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p0oRjXomt9hR5NylVHVEYf6Xzyavks5sJ725gaJpZM4HGVRn> .

thockin · 2017-07-04T17:59:04Z

@pmorie and I had sketched out a "merge" volume, but we'd need an ower On Sun, Jul 2, 2017 at 7:38 PM, Clayton Coleman <notifications@github.com> wrote:

…

I cry a little bit every time I hit this limitation. It's definitely one of my top 10 usability issues for any complex pod (since a lot of Unix software uses `/etc/` and we can't project into `/etc` without a hammer or hacks. On Sun, Jul 2, 2017 at 11:56 AM, Tim Hockin ***@***.***> wrote: > This is not specific to configmap but to any subpath projection. > > On Jul 2, 2017 5:37 AM, "Boyko Alexey" ***@***.***> wrote: > > > @schmitzhermes <https://github.com/schmitzhermes> unfortunately, as I > can > > see, configmap can't be updated when subPath is defined :( > > > > — > > You are receiving this because you modified the open/close state. > > Reply to this email directly, view it on GitHub > > <https://github.com/kubernetes/kubernetes/issues/ > 19764#issuecomment-312489424>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/AFVgVC5eqQSuicV6- > HuDwr88aDsC2D6wks5sJ48jgaJpZM4HGVRn> > > . > > > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/kubernetes/issues/ 19764#issuecomment-312500042>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABG_ p0oRjXomt9hR5NylVHVEYf6Xzyavks5sJ725gaJpZM4HGVRn> > . > — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#19764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVEz5ixnRvQura145oqjmnEX8fWYYks5sKFQdgaJpZM4HGVRn> .

funlake · 2018-03-26T08:19:25Z

+1 cry every time,any plans for single file volume mapping?

thockin · 2018-03-31T01:19:48Z

There are more feature requests than people to implement. ;(

…

On Mon, Mar 26, 2018, 1:19 AM funlake ***@***.***> wrote: +1 cry every time,any plans for single file volume mapping? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#19764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVPmOkoLAwJFnLl8PbjNClV0NOiyMks5tiKSlgaJpZM4HGVRn> .

Chrislevi · 2018-08-28T01:38:15Z

Any news? hit the same wall

SleepyBrett · 2018-08-28T14:41:29Z

This becomes an even bigger issue when you widely enable restricted psps that don't allow for rootfs writes.... which we did last week.

In this case I need to mount hostpath:/etc/machine-id into container:/etc/machine-id ... I clearly can't just replace container:/etc w/ hostpath:/etc

thockin · 2018-08-28T15:16:21Z

I'd love to see this get worked on, but it needs an owner. I'm happy to reopen if someone wants to tackle it..

ffledgling · 2019-03-25T00:52:07Z

@thockin what would implementing this take? I saw you mention in one of the related issues that it's technically hard to do.

pmorie added area/api Indicates an issue on api area. area/volumes labels Jan 16, 2016

bgrant0607 added the team/cluster label Jan 21, 2016

bgrant0607 added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Mar 7, 2016

bgrant0607 added this to the next-candidate milestone Mar 7, 2016

bgrant0607 added the area/usability label Mar 7, 2016

saad-ali added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Mar 7, 2016

bgrant0607 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/friction and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Mar 7, 2016

thockin mentioned this issue Mar 19, 2016

Proposal: Mount each secret as a file, not as a directory #23237

Closed

donspaulding mentioned this issue Jan 25, 2017

Better Secrets management workflow kubernetes/dashboard#1593

Closed

tkaymak mentioned this issue Feb 19, 2017

Updating dags mumoshu/kube-airflow#1

Closed

bitmage mentioned this issue Mar 30, 2017

Can't map two secrets to /etc/secrets ployst/docker-nginx-ssl-proxy#9

Open

bitmage mentioned this issue Apr 1, 2017

Symlink in writable volume overwritten #43938

Closed

wongma7 mentioned this issue Apr 10, 2017

[nfs-provisionner] Unable to share single files kubernetes-retired/external-storage#69

Closed

deitch mentioned this issue Jul 5, 2017

Proposal: syntax for loading data into containers linuxkit/linuxkit#2151

Open

liggitt mentioned this issue Oct 6, 2019

In Windows, configMap or Secret volumes might overwrite existing dir in container #83511

Closed

Projecting single files from volumes into a container's FS #19764

Projecting single files from volumes into a container's FS #19764

Comments

pmorie commented Jan 16, 2016

pmorie commented Jan 17, 2016

thockin commented Jan 17, 2016

mfojtik commented Jan 17, 2016

mfojtik commented Jan 17, 2016

thockin commented Jan 17, 2016

bgrant0607 commented Jan 21, 2016

pmorie commented Jan 21, 2016

thockin commented Jan 22, 2016

pmorie commented Jan 22, 2016

smarterclayton commented Jan 22, 2016

thockin commented Jan 22, 2016

erictune commented Feb 12, 2016

thockin commented Feb 12, 2016

kelseyhightower commented Mar 7, 2016

thockin commented Mar 7, 2016

eparis commented Mar 7, 2016

smarterclayton commented Mar 8, 2016 via email

pmorie commented Mar 8, 2016

thockin commented Mar 19, 2016

deitch commented Feb 19, 2017

deitch commented Feb 19, 2017

bryanlarsen commented Mar 2, 2017

bitmage commented Mar 30, 2017

deitch commented Mar 30, 2017

thockin commented Mar 31, 2017

debianmaster commented Apr 16, 2017

debianmaster commented Apr 16, 2017

fearphage commented May 1, 2017

schmitzhermes commented May 26, 2017

debianmaster commented May 26, 2017

ketchoop commented Jul 2, 2017

thockin commented Jul 2, 2017 via email

smarterclayton commented Jul 3, 2017 via email

thockin commented Jul 4, 2017 via email

funlake commented Mar 26, 2018

thockin commented Mar 31, 2018 via email

Chrislevi commented Aug 28, 2018

SleepyBrett commented Aug 28, 2018 • edited

thockin commented Aug 28, 2018

ffledgling commented Mar 25, 2019

SleepyBrett commented Aug 28, 2018 •

edited