Dockerfile: ADD does not honor USER: files always owned by root

[discordianfish]

Hi,

consider this Dockerfile:

FROM ubuntu
RUN adduser foo
USER foo
ADD . /foo

/foo in the container will be owned by root, not by 'foo' as one might expect. There are easy workaround but it's still a inconsistency which should be fixed at some point.

[LK4D4]

Cool, I'll do it on weekend :)

[crosbymichael]

FYI, I'm not sure we want this behavior, we will need to discuss first

[abevoelker]

I'm relatively new to Docker and this had confused me at first as well. Every time I have to add files that should belong to a non-root user, I have to do RUN chown after the ADD (keeping it all above the USER line since chown needs root permission).

I had just assumed it was expected behavior and there was a good reason for it. The documentation for ADD does make it sound like expected behavior:

All new files and directories are created with mode 0755, uid and gid 0.

[LK4D4]

Also it was in original issue about chowning, that have been done by @creack. But I was too lazy to do it and @creack also didn't do it by some reasons.
#2684 - original issue

[unclejack]

I don't think making ADD set file ownership to whatever user the USER instruction has set above is a good idea. USER was initally added to specify the user which the commands executed by RUN or CMD should be run under.

[discordianfish]

@unclejack Why do you think this isn't a good idea? I think it's what the user expects: All following operations will be executed as the given user.

[divoxx]

I just got bitten by this and gotta say it's a surprise that ADD does not respect the preceding USER declaration. If someone wants to add as root, simply run ADD before the USER is set to something else, as one would do w/ RUN.

+1 for this

[discordianfish]

Keep in mind we're recommend that people run their services as a non-privileged user. Inconveniences like this behavior will prevent people from following that.

[crosbymichael]

Are there any issues running a chown after you add things?

Ref: https://github.com/crosbymichael/influxdb-docker/blob/master/Dockerfile#L9

[tianon]

Doesn't AUFS have issues with tracking ownership/mode changes?

[crosbymichael]

@tianon yes

[LK4D4]

@crosbymichael yeah, this is very inconvenient. for example I have postgres image, there is already USER postgres in it, so I need to do:

USER root
RUN chown
USER postgres

Also I must know about USER postgres in base image.

[pikeas]

Proposal: a second form of the ADD command. I'd love to be able to do:

ADD ["filename", "owner", "group", "755"]

[cyphar]

I open a proposal a few days ago that addresses this issue: #7537. Essentially, after talking on IRC, it was decided that only COPY should have this particular behaviour (and ADD will be backward compatible until it is deprecated and removed).

[ericchaves]

Hi guys, any update on this issue?

[david-mccullars]

Being forced to ADD and then RUN chown has a very nasty side-effect of artificially bloating docker images. For example:

FROM busybox:latest

RUN adduser -D bob
ADD 30MB_file /30MB_file
RUN chown bob /30MB_file

I ran build on this once without the chown and once with:

➔ docker images | grep bob
test    with_chown          a27913861eca        6 days ago          65.35 MB
test    without_chown       7ab2bbe29b50        6 days ago          33.89 MB

And we can see what is happening via docker history:

 ➔ docker history test:with_chown
IMAGE         CREATED        CREATED BY                                      SIZE
a27913861eca  6 days ago     /bin/sh -c chown bob /30MB_file                 31.46 MB
7ab2bbe29b50  6 days ago     /bin/sh -c #(nop) ADD file:9474e987e88bd93248   31.46 MB
e9413ccecaa1  6 days ago     /bin/sh -c adduser -D bob                       3.955 kB
4986bf8c1536  2 weeks ago    /bin/sh -c #(nop) CMD [/bin/sh]                 0 B
ea13149945cb  2 weeks ago    /bin/sh -c #(nop) ADD file:8cf517d90fe79547c4   2.433 MB
df7546f9f060  3 months ago   /bin/sh -c #(nop) MAINTAINER Jérôme Petazzo     0 B
511136ea3c5a  19 months ago                                                  0 B

When we consider that a large application might have several hundred megabytes or more, this chown becomes very expensive.

[cyphar]

#9934 has been opened to fix this issue (my patches were rejected).

[goldmann]

This behavior really, really needs to be changed to make sure ADD uses current USER. Current way of implementing it with chown:

1. Created 3 (sic!) unnecessary layers
2. Makes Dockerfile ugly
3. Is totally not intuitive.

[sandric]

@goldmann agree on all points!

[thaJeztah]

This topic was discussed by the maintainers and, while they agreed that having USER specify the owner of files added by ADD / COPY would have been a better choice, changing this behavior now would be a breaking change, which would be too much of a risk.

For this reason, #9934 was created to come with alternatives. An implementation of this is created in #10775, which is currently under review.

For those interested, please follow the review/discussion on #9934 to check progress.

[duglin]

@kostickm is working on a proposal for this.

[kiwenlau]

I think fix this problem will help to decrease images size. Specify users when ADD files is a great idea!

For example, if I ADD a binary like Hadoop, then chown it to hadoop user, the image size will increase 100+MB!

That's why I decided to install hadoop under root user:(

[borntorock]

Also, if someone wants to change the owner and group owner of the files added inside the docker image via COPY or ADD, that actually increases the size of the image by the size of the files added.

It would be great to have COPY2 so that the USER with whom we're actually copying the files inside the docker image becomes the owner and groupowner.

[rhuss]

I'm subscribed on this issue since maybe two years, and I'm really puzzled why no solution has been found, whatever this looks like

One of the first arguments was that the Dockerfile format is frozen. That might have been true at that time, but does not hold anymore as there as been some changes since then (like FROM ... AS or adding HEALTHCHECK)

Since the last release the option --from has been added to COPY. So there is also no reason anymore against adding options to Dockerfile keywords.

So what are the arguments against a COPY --chown <uid>:<gid> these days?

[attila-lendvai]

@thaJeztah i tried to be ironic about the developers lamenting that fixing this bug would be a breaking change. no shit, who would have thought!

just the amount of discussion that happens in the numerous threads that i encountered is a lot of time wasted, both by the developers and by the users. and all that for what? trying to protect some hypothetical users who knowingly relied on the behavior that COPY, rather surprisingly, ignores USER?

what are major version numbers for? and release notes?

if there actually exists anyone who relies on this bug then they should not upgrade to a new major release, or read the release notes.

and let's say their code is broken because they didn't read the release notes... what about the tons of users who get bitten by this every single day? maybe one outweighs the other...

[dvdgsng]

So, this is considered to be a breaking change while renaming the whole project is not?

[goldmann]

This was commited a few months ago: 0725045. I'll leave it just here.

[DavidAntliff]

Well, just to vent my frustration - this just caught me out (and I've been using Docker for a year now). How is this still a problem?

[softwareplumber]

Wow. I'm a complete Docker newb and just ran into this one. From my virgin perspective, I absolutely expected ADD and COPY to respect USER. That they don't was a source of considerable astonishment that increased exponentially when I saw just how long ago this issue was raised and how simple the apparent fix is. I don't generally post to zombie threads, but felt obliged to add my insignificant voice to the general chorus of bafflement expressed here.

[italomaia]

This issue was closed because ... ?

[ehernandez-xk]

As a workaround using Alpine for me it was:

• COPY xfile location/xfile
• RUN adduser -S xuser
• RUN chown xuser -R location/xfile
• USER xuser

[borntorock]

@ehernandez-xk Hey, Looks like you didn't read the previous comments. Though this seems to be a workaround, but this is going to increase your image size by the size of your xfile.

what if incase the files I'm copying inside the docker image is of 1Gb. I can't afford to have my image size increased by 1GB.

[borntorock]

@italomaia Not sure, why this issue was closed?

@thaJeztah This is not fixed.

[robhaswell]

what if incase the files I'm copying inside the docker image is of 1Gb. I can't afford to have my image size increased by 1GB.

It also may take an unacceptably long amount of time.

[italomaia]

Hey @jessfraz is the reason why this issue was closed still valid?

[softwareplumber]

Perhaps there is some way to figure out how much storage space is being used on AWS to store docker FS layers that do nothing but change ownership of a previously-copied resource, and offer a bounty based upon the cost of such storage... Who knows, maybe Amazon would like a few terabytes of storage back...

…

On Jun 8, 2017, at 5:24 PM, David Antliff ***@***.***> wrote: As with other issues, round and round we go. The fact that no Docker developer has reopened this speaks volumes about the way the Docker project is run. This is a common criticism of Docker's developers and it's a red flag for anyone thinking of building production infrastructure on Docker. I recommend looking at other options, as this situation doesn't look like it's going to improve. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYDy42P0kzpN4ALmiOSlagXjrfvm64JPks5sCGadgaJpZM4B_uID>.

[DeRain]

@ehernandez-xk @borntorock it seems we could use --squash as a workaround simultaneously with chown

[karneaud]

+1

[softwareplumber]

Personally, my biggest problem with the status quo is the amount of time the separate chown operation takes, as is seems to be copying the files as it changes ownership. Adding squash would presumably just make this worse...

…

On Jun 12, 2017, at 2:11 PM, Kirill Beresnev ***@***.***> wrote: @ehernandez-xk <https://github.com/ehernandez-xk> @borntorock <https://github.com/borntorock> it seems we could use --squash as a workaround simultaneously with chown — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYDy42WPKw3o-dksFpTnMulbqyASLmtqks5sDSsHgaJpZM4B_uID>.

[thiagowfx]

TL;DR non-intuitive behavior; use chown -R to set / enforce permissions after ADD or COPY if you want them to be other than root; USER has no effect on file permissions whatsoever.

[paulp]

@thiagowfx Your tl;dr summarizes a little too aggressively - it bears mentioning that this will double the size of your docker image.

[softwareplumber]

The additional build time is also a concern. For me at least, when adding a large archive, the chown -R seems to take longer than the original ADD operation.

…

On Jul 11, 2017, at 4:32 PM, Paul Phillips ***@***.***> wrote: @thiagowfx <https://github.com/thiagowfx> Your tl;dr summarizes a little too aggressively - it bears mentioning that this will double the size of your docker image. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYDy45yLHw__ewUuVJez78OrPBT5dYJbks5sM9vggaJpZM4B_uID>.

[DavidAntliff]

It's typically worse than just a chown. Because you're likely to be "running" as a non-root user (which is why you probably encountered this issue in the first place), you need to change the root user first, and then switch back to your non-root user afterwards. It typically results in four commands:

COPY ...
USER root
RUN chown -R ...
USER <original>

It can be improved by allowing your user to run sudo /bin/chown in /etc/sudoers (one command) but that has a bad smell to it in my opinion (not to mention a security hole if you are locking the container down in other ways), so you add another line to remove it later (and then the next layer adds it back again to do the same thing...)

Surely something like the following is better for everyone?

COPY --owner user.group

Can the Docker Engine plugin interface be used to implement new commands such as this?

[italomaia]

There is already a PR for this particular problem, but it is closed, waiting for some discussions to end. #28499

[andyburton]

This is extremely painful when having to do an npm install with COPY followed by chown, and combined with the Mac filesystem performance issues! A simple COPY with chown would make a significant difference.

[kaikuchn]

To save those who are interested in seeing this fixed a lot of time: here is the current attempt to implement a fix.

[JPvRiel]

Wow. I'm a complete Docker newb and just ran into this one. From my virgin perspective, I absolutely expected ADD and COPY to respect USER.

Agree, this is not intuitive. USER was added to set the context for RUN and CMD instructions, and so one might assume it's more or less like an su USER and should also apply to COPY/ADD.

However, under the hood, docker is running itself and the build as root? So if root is the process doing the copy, then the resulting file is also owned by root. E.g. In an ordinary shell, when a cp command is executed, the UNIX permissions of the process running the copy command is used by default (unless special options are used). So docker implemented a pseudo user context switch, but only for some instructions and not others? This boils down to being somewhat inconsistent. Its confusing to have a user context only imply half the time...

[thaJeztah]

This feature has now been implemented through an optional --chown flag on ADD and COPY through pull request: #34263

The feature is included in Docker 17.09 and up, and allows you to do;

ADD --chown=someuser:somegroup /foo /bar
COPY --chown=someuser:somegroup /foo /bar

Or other combinations of user/group name (or ID);

--chown=someuser:123
--chown=anyuser:anygroup
--chown=1001:1002
--chown=333:agroupname

There are two additional feature requests in this area;

• Allow COPY command's --chown to be dynamically populated via ENV or ARG #35018 ("Allow COPY command's --chown to be dynamically populated via ENV or ARG")
• Proposal: Add --chmod flag to ADD/COPY commands (analogous to --chown) #34819 ("Proposal: Add --chmod flag to ADD/COPY commands (analogous to --chown)")

Given that the original problem was addressed (the implementation is different from the one proposed in this issue, you can read the discussion in #9934 and #9934 (comment) for more background on that), I'm locking the discussion on this issue to prevent it from collecting other (possibly unrelated) comments.

If you arrive on this issue because this feature is not working as expected, or you think other enhancements can be made; please open a new issue instead so that it can be tracked separately (feel free to add a link to this issue if you think it's relevant).