New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --chown flag to ADD/COPY commands #34263

Merged
merged 2 commits into from Aug 28, 2017

Conversation

@estesp
Contributor

estesp commented Jul 26, 2017

Carry #28499 (which was a carry of #27303)

This adds --chown function to the ADD and COPY commands in Dockerfile. The format of the --chown flag allows any uid/username and gid/groupname combination separated by a colon, e.g.:

--chown=someuser:555
--chown=anyuser:anygroup
--chown=1001:1002
--chown=333:agroupname

Lookups (to translate name -> ID integer) will use the /etc/passwd and /etc/group files found in the container filesystem. Currently if they do not exist, then the docker build will fail. This PR probably needs a way to fail gracefully (or ignore failures) on Windows as those containers will not have these files. If only numeric IDs are used, no lookups will be performed.

User namespaces are supported and IDs will be shifted (after translated names to integers) based on the user namespace mappings set in the daemon.

I also think a few more tests are necessary to fully test the behavior, but wanted to get the rebased code available for review initially.

Add --chown flag to Dockerfile ADD and COPY
Rebased by @estesp

Signed-off-by: Kara Alexandra <kalexandra@us.ibm.com>
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Show outdated Hide outdated builder/dockerfile/copy.go Outdated
@CWSpear

This comment has been minimized.

Show comment
Hide comment
@CWSpear

CWSpear Jul 27, 2017

Contributor

This was quite a journey. I Googled my issue (the whole "having to chown" stuff) and found an issue from 2 years ago.

Then I had to keep reading through massive threads, with both Docker contributors and the community supporting this, but kept having to pick out "continuing this thread [here]," and about 7 or 8 threads later, I find this, which is obviously a very recent PR.

So... where are we with all this? :)

Contributor

CWSpear commented Jul 27, 2017

This was quite a journey. I Googled my issue (the whole "having to chown" stuff) and found an issue from 2 years ago.

Then I had to keep reading through massive threads, with both Docker contributors and the community supporting this, but kept having to pick out "continuing this thread [here]," and about 7 or 8 threads later, I find this, which is obviously a very recent PR.

So... where are we with all this? :)

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Jul 27, 2017

Contributor

Hopefully as close as it has ever been to reality! With the maintainers discussion and comments (like this one from issue #30110 I would expect it is just a matter of review and cleaning up a few things with my implementation.

Given @justincormack wants it, I assume he will be a huge supporter and reviewer of my PR, right Justin! =)

Contributor

estesp commented Jul 27, 2017

Hopefully as close as it has ever been to reality! With the maintainers discussion and comments (like this one from issue #30110 I would expect it is just a matter of review and cleaning up a few things with my implementation.

Given @justincormack wants it, I assume he will be a huge supporter and reviewer of my PR, right Justin! =)

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jul 28, 2017

Contributor

Some minor comments but otherwise looks good. I think it is ok to fail on named users on Windows for now, maybe have a test for platform rather than trying to look for /etc/passwd.

Contributor

justincormack commented Jul 28, 2017

Some minor comments but otherwise looks good. I think it is ok to fail on named users on Windows for now, maybe have a test for platform rather than trying to look for /etc/passwd.

Show outdated Hide outdated builder/dockerfile/copy.go Outdated
Show outdated Hide outdated builder/dockerfile/copy.go Outdated
Show outdated Hide outdated builder/dockerfile/dispatchers.go Outdated
Show outdated Hide outdated builder/dockerfile/dispatchers.go Outdated
Show outdated Hide outdated builder/dockerfile/internals.go Outdated
Show outdated Hide outdated builder/dockerfile/internals.go Outdated
Show outdated Hide outdated builder/dockerfile/internals.go Outdated
return users[0].Uid, nil
}
func lookupGroup(groupStr, filepath string) (int, error) {

This comment has been minimized.

@dnephin

dnephin Jul 28, 2017

Member

I think this already exists as idtools.LookupGroup() / LookupGID()? Can we re-use either of those?

@dnephin

dnephin Jul 28, 2017

Member

I think this already exists as idtools.LookupGroup() / LookupGID()? Can we re-use either of those?

This comment has been minimized.

@estesp

estesp Jul 28, 2017

Contributor

No

@estesp

estesp Jul 28, 2017

Contributor

No

Show outdated Hide outdated builder/dockerfile/internals.go Outdated
@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jul 28, 2017

Member

Looks like windows failures are real because of lchown. I think maybe fixed by removing the code I suggested is not necessary.

Member

dnephin commented Jul 28, 2017

Looks like windows failures are real because of lchown. I think maybe fixed by removing the code I suggested is not necessary.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Jul 28, 2017

Contributor

Yes, the Windows failures are real--my delay in properly handling here yet is on the design point of whether to fail or silently ignore; seems ignore (by using some sort of _windows.go variation of the lookups/chown codepaths) is probably better if you can see a case for Dockerfile sharing, but that seems unlikely in lots of cases anyway. Do we want to fail on a --chown flag on ADD/COPY on Windows?

Contributor

estesp commented Jul 28, 2017

Yes, the Windows failures are real--my delay in properly handling here yet is on the design point of whether to fail or silently ignore; seems ignore (by using some sort of _windows.go variation of the lookups/chown codepaths) is probably better if you can see a case for Dockerfile sharing, but that seems unlikely in lots of cases anyway. Do we want to fail on a --chown flag on ADD/COPY on Windows?

@shouze

This comment has been minimized.

Show comment
Hide comment
@shouze

shouze Jul 31, 2017

Contributor

@estesp @dnephin can/will this PR fix #32816?

Contributor

shouze commented Jul 31, 2017

@estesp @dnephin can/will this PR fix #32816?

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Aug 10, 2017

Contributor

@dnephin and others; PR has been updated. A bit of vacation delay, sorry :)

At this point I believe the only review comments not handled is the recommended consolidation of pkg/idtools parsing of host /etc/{passwd,group} files and the new parsing here of container specific /etc/ files.

After looking through the idtools package use of the opencontainers/runc/libcontainer/user package and this new code's use, I don't see any effective way of combining these without simply adding complexity. In essence, the user package offers two paths:

  1. the "easy" default path of letting that package read host default files; and
  2. the more complex path of providing that code filepaths or open readers along with a custom filter function to find a specific entry.

The idtools package uses the "easy" path, because that fits the need of that package, and this new PR code uses the other path, because we want to reference specific paths inside the container we're using during build. Unless we force idtools to take the more complex path, and basically rewrite the "easy path" as a consumer of the functions already in that user package. Either way we end up duplicating code by copying functions that currently exist in user. I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

Contributor

estesp commented Aug 10, 2017

@dnephin and others; PR has been updated. A bit of vacation delay, sorry :)

At this point I believe the only review comments not handled is the recommended consolidation of pkg/idtools parsing of host /etc/{passwd,group} files and the new parsing here of container specific /etc/ files.

After looking through the idtools package use of the opencontainers/runc/libcontainer/user package and this new code's use, I don't see any effective way of combining these without simply adding complexity. In essence, the user package offers two paths:

  1. the "easy" default path of letting that package read host default files; and
  2. the more complex path of providing that code filepaths or open readers along with a custom filter function to find a specific entry.

The idtools package uses the "easy" path, because that fits the need of that package, and this new PR code uses the other path, because we want to reference specific paths inside the container we're using during build. Unless we force idtools to take the more complex path, and basically rewrite the "easy path" as a consumer of the functions already in that user package. Either way we end up duplicating code by copying functions that currently exist in user. I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

@g13013

This comment has been minimized.

Show comment
Hide comment
@g13013

g13013 Aug 13, 2017

Any news on that ?

g13013 commented Aug 13, 2017

Any news on that ?

Show outdated Hide outdated builder/dockerfile/internals.go Outdated
Show outdated Hide outdated integration-cli/docker_cli_build_test.go Outdated
return chownPair, nil
}
func lookupUser(userStr, filepath string) (int, error) {

This comment has been minimized.

@dnephin

dnephin Aug 15, 2017

Member

This (and lookupGroup) should have unit tests for each case

@dnephin

dnephin Aug 15, 2017

Member

This (and lookupGroup) should have unit tests for each case

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Aug 15, 2017

Member

I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

Thanks for looking into it

Member

dnephin commented Aug 15, 2017

I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

Thanks for looking into it

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Aug 16, 2017

Contributor

@dnephin bugs created during the re-work of those functions are fixed and unit tests added that test all formats of the input string and with/without userns mappings. This tests the parseChownFlag and also the lookupUser/lookupGroup functions. Let me know if you see any holes there.

I understand the comments re: testing, but as you note integration is barely started, and I don't see any movement of cli_* content to api_* yet. I could see potentially adding this to build API, but that will take some extra work as there is very poor scaffolding for standard use of test files, etc. Before I do that can we resolve the ADD and COPY discussion? I wasn't in the maintainer's meeting where it was recommended this work proceed, so I haven't been given any clue as to whether there as a debate around just COPY vs. COPY + ADD. If you mean only one test given the code paths are very similar, that makes sense to me.

Contributor

estesp commented Aug 16, 2017

@dnephin bugs created during the re-work of those functions are fixed and unit tests added that test all formats of the input string and with/without userns mappings. This tests the parseChownFlag and also the lookupUser/lookupGroup functions. Let me know if you see any holes there.

I understand the comments re: testing, but as you note integration is barely started, and I don't see any movement of cli_* content to api_* yet. I could see potentially adding this to build API, but that will take some extra work as there is very poor scaffolding for standard use of test files, etc. Before I do that can we resolve the ADD and COPY discussion? I wasn't in the maintainer's meeting where it was recommended this work proceed, so I haven't been given any clue as to whether there as a debate around just COPY vs. COPY + ADD. If you mean only one test given the code paths are very similar, that makes sense to me.

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Aug 17, 2017

Member

here is very poor scaffolding for standard use of test files

Not true. All the same scaffolding works for the api tests. See https://github.com/moby/moby/blob/master/integration-cli/docker_api_build_test.go#L288-L306

I don't see any movement of cli_* content to api_*

There should be. Every PR I have seen has called out to change the cli test to an api test

Member

dnephin commented Aug 17, 2017

here is very poor scaffolding for standard use of test files

Not true. All the same scaffolding works for the api tests. See https://github.com/moby/moby/blob/master/integration-cli/docker_api_build_test.go#L288-L306

I don't see any movement of cli_* content to api_*

There should be. Every PR I have seen has called out to change the cli test to an api test

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Aug 17, 2017

Member

If you mean only one test given the code paths are very similar, that makes sense to me.

Yes, I think a single test is sufficient.

@tonistiigi Do you think we should add this flag to ADD as well? or just COPY ?

Member

dnephin commented Aug 17, 2017

If you mean only one test given the code paths are very similar, that makes sense to me.

Yes, I think a single test is sufficient.

@tonistiigi Do you think we should add this flag to ADD as well? or just COPY ?

@dnephin

Thanks, unit tests are looking good, just a couple comments

Show outdated Hide outdated builder/dockerfile/internals_test.go Outdated
Show outdated Hide outdated builder/dockerfile/internals_test.go Outdated

@estesp estesp requested a review from vdemeester as a code owner Aug 17, 2017

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Aug 17, 2017

Contributor

@dnephin thanks for the review; I've updated the unit tests with your recommendations and moved a single "COPY"-based test into the docker_api_ build tests from docker_cli_build_* and removed the two tests that were there. PTAL.

Contributor

estesp commented Aug 17, 2017

@dnephin thanks for the review; I've updated the unit tests with your recommendations and moved a single "COPY"-based test into the docker_api_ build tests from docker_cli_build_* and removed the two tests that were there. PTAL.

@vincentwoo

This comment has been minimized.

Show comment
Hide comment
@vincentwoo

vincentwoo Oct 26, 2017

Contributor

I saw this change in the release log for the latest stable version of docker-ce, but it doesn't appear to be in the docs.

Is this actually released?

Contributor

vincentwoo commented Oct 26, 2017

I saw this change in the release log for the latest stable version of docker-ce, but it doesn't appear to be in the docs.

Is this actually released?

@mwarkentin

This comment has been minimized.

Show comment
Hide comment
@mwarkentin

mwarkentin Oct 26, 2017

@vincentwoo yes, it’s working in docker 17.09-ce.

mwarkentin commented Oct 26, 2017

@vincentwoo yes, it’s working in docker 17.09-ce.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 26, 2017

Member

See docker/cli#467 for the docs; that PR was a bit delayed 😅

Member

thaJeztah commented Oct 26, 2017

See docker/cli#467 for the docs; that PR was a bit delayed 😅

@vipseixas

This comment has been minimized.

Show comment
Hide comment
@vipseixas

vipseixas Nov 13, 2017

--chown does not work for compressed files (eg: tgz), it seems to only change the owner of the compressed file itself and then ignore the extracted content.

Is it the intended behavior?

vipseixas commented Nov 13, 2017

--chown does not work for compressed files (eg: tgz), it seems to only change the owner of the compressed file itself and then ignore the extracted content.

Is it the intended behavior?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 13, 2017

Member

@vipseixas I'd say; yes, because it'll only change permissions on the file you add. Archives themselves can have permissions set as well, which are preserved (if I'm not mistaken).

Given that automatically extracting archives only works for local files; can you explain your use-case?
i.e., instead of compressing local files, adding them to the image, and then decompressing, why not add the uncompressed files directly?

Member

thaJeztah commented Nov 13, 2017

@vipseixas I'd say; yes, because it'll only change permissions on the file you add. Archives themselves can have permissions set as well, which are preserved (if I'm not mistaken).

Given that automatically extracting archives only works for local files; can you explain your use-case?
i.e., instead of compressing local files, adding them to the image, and then decompressing, why not add the uncompressed files directly?

@vipseixas

This comment has been minimized.

Show comment
Hide comment
@vipseixas

vipseixas Nov 13, 2017

@thaJeztah I have a 3rd party installation (PHP code) that comes as a 180MB tar.gz file and I have to open it inside Apache's /var/www. The uncompressed files grows to ~570MB so it would be easier to store the compressed file with my Dockerfile. But if I ADD the compressed file and use a "RUN chown" I end up with 2 huge layers in my image.

vipseixas commented Nov 13, 2017

@thaJeztah I have a 3rd party installation (PHP code) that comes as a 180MB tar.gz file and I have to open it inside Apache's /var/www. The uncompressed files grows to ~570MB so it would be easier to store the compressed file with my Dockerfile. But if I ADD the compressed file and use a "RUN chown" I end up with 2 huge layers in my image.

@derikson

This comment has been minimized.

Show comment
Hide comment
@derikson

derikson Nov 13, 2017

@vipseixas as a workaround, you could use multi-stage build to extract the files in one build stage, then COPY with chown in the second stage. This would prevent the final image from having the extra layer.

derikson commented Nov 13, 2017

@vipseixas as a workaround, you could use multi-stage build to extract the files in one build stage, then COPY with chown in the second stage. This would prevent the final image from having the extra layer.

@vipseixas

This comment has been minimized.

Show comment
Hide comment
@vipseixas

vipseixas Nov 13, 2017

@derikson Your suggestion worked perfectly! I was not aware of this new feature, thanx.

vipseixas commented Nov 13, 2017

@derikson Your suggestion worked perfectly! I was not aware of this new feature, thanx.

@marius311

This comment has been minimized.

Show comment
Hide comment
@marius311

marius311 Nov 17, 2017

This is awesome, thanks guys! Probably one of my most desired features for years.

At the risk of being one those never-satisfied Docker users though :) ...

@thaJeztah:

It's not a bug; variable expansion in options is not supported at this moment

Is this planned at all and if so is there an issue we could track this at?

marius311 commented Nov 17, 2017

This is awesome, thanks guys! Probably one of my most desired features for years.

At the risk of being one those never-satisfied Docker users though :) ...

@thaJeztah:

It's not a bug; variable expansion in options is not supported at this moment

Is this planned at all and if so is there an issue we could track this at?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 17, 2017

Member

Is this planned at all and if so is there an issue we could track this at?

A feature request for that was opened here: #35018 (but not decided on yet 😅)

Member

thaJeztah commented Nov 17, 2017

Is this planned at all and if so is there an issue we could track this at?

A feature request for that was opened here: #35018 (but not decided on yet 😅)

@khatribharat

This comment has been minimized.

Show comment
Hide comment
@khatribharat

khatribharat Dec 27, 2017

This isn't yet reflected in the Dockerfile doc

khatribharat commented Dec 27, 2017

This isn't yet reflected in the Dockerfile doc

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 27, 2017

Member

@khatribharat docs took some time to get merged (see docker/cli#467), but will be updated once Docker 17.12 has been released

Member

thaJeztah commented Dec 27, 2017

@khatribharat docs took some time to get merged (see docker/cli#467), but will be updated once Docker 17.12 has been released

@thaJeztah thaJeztah referenced this pull request Jan 26, 2018

Closed

Unknown flag: chown #35731

aglorei added a commit to aglorei/pokemon_battle_rails that referenced this pull request Feb 2, 2018

Use new COPY --chown flag
Ever since moby/moby/pull/34263 became merged, there's no need to
shuffle users in order to get the correct ownership on non-root users

@omercnet omercnet referenced this pull request Feb 26, 2018

Closed

Add --chown to WORKDIR #36408

sinhyeok-cp added a commit to EOSYS-io/eoshub.io that referenced this pull request Jun 27, 2018

@gucharbon gucharbon referenced this pull request Sep 12, 2018

Open

301 docker #318

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment