Dockerfile 'ADD' to support git repositories

[yaronr]

It would be cool if a Dockerfile command such as ADD could support git sources, something like:

ADD https://github.com/something/something.git
(at least for public git repos, maybe even with a few arguments - for private ones as well)

Just a thought

[ewindisch]

I could see this being offered, potentially, for git:// scheme URLs. I'm not sure it makes sense to support this for http/https URLs, unless formatted as "https+git://".

[dspaxton]

I'd like to add support for this if possible as would reduce the need to include git within the image.

[yaronr]

@numanoids Exactly where I'm coming form. I think that this is relevant for a lot of cases.

[cpuguy83]

The problem here is that it'd pretty much have to be https since there won't be an ssh key available to clone over SSH.

[thaJeztah]

Just FYI, we currently don't accept changes to the Dockerfile syntax (and behavior), until after the builder has been moved to the client. (see the roadmap)

Discussing it shouldn't hurt though.

[dkirrane]

It would be good also if ADD https://github.com/something/something.git used the cache and only cloned the repo again if the git SHA hashcode changed from the previous docker build

[mauermbq]

you may use oAuth token to automatically download a git repo

[tzz]

@yaronr branch/tag, clone depth, and other options would be useful. So maybe this needs to be a new top-level command ADDREPO with many of the options that git clone provides (and indeed, it could just call that command in the end). Git is also not the only VCS out there; at least Subversion and Mercurial should also have some support, so it should be extensible.

@cpuguy83 maybe the SSH agent could be passed down to the build process? The alternatives (embed SSH key in the image or use a deployment/oAuth token) that people are using are pretty nasty.

@dkirrane agreed! Downloading the whole repo would be painful.

[rainbreak]

@dkirrane it is possible to do this now, by using the github api to query the commits on a branch:

ADD https://api.github.com/repos/docker/docker/compare/master...HEAD /dev/null
RUN git clone https://github.com/docker/docker

This will only clone e.g. docker/docker again if the latest commit on the master branch has changed. See https://developer.github.com/v3/repos/commits/ for more info.

This doesn't solve the problem of requiring git, but it does address the caching problem.

[schovi]

@rainbeam Just found this solution and I wonder how this works? How docker cache RUN command depends on result of that ADD?

[unclejack]

This has been discussed before on a few different issues. Implementing git support for ADD means that we'd have to add support for subversion, mercurial and a few other SCM systems. This would also make the Dockerfile less portable. A version of git might be newer on one host than on another one. Different versions compiled against different libraries could have different behaviors.
The current way to run git clone in RUN isn't perfect, but at least it behaves the same way on different hosts. One way to avoid this problem would be to run this git clone operation inside a container.

Another problem is that ADD has a lot of magic in it. It'd be a good idea to not overload it any further. It's not what some would want to hear. The right thing to do here would be to take into account a new Dockerfile instruction which would use a container to clone repositories. This could also make it possible to implement support for different SCM systems.

[aeijdenberg]

Hi all, I found this bug when looking for something similar to help one of my clients out who is looking to eventually build Docker images as part of their CI process, which will involve pulling from 3 different git repos to combine code, config and some other scaffolding.

After reading this bug, particularly the last comment by @unclejack, I put together a proof-of-concept patch that allows for one Dockerfile to kick off a build of a second image, and then "cherry-pick" a copy of the final layer it produces back onto itself.

In this manner we have a somewhat generic mechanism to get output from one Dockerfile for use in building another.

Here's a worked example demonstrating the patch:

Let's say I create a Dockerfile that can be used to fetch Go source for a particular project and build it - we'll base it on the standard golang one:

$ mkdir GoBuild
$ cat <<'EOF' > GoBuild/Dockerfile
FROM golang:1.7
ARG gosrc
RUN go get ${gosrc}
RUN cp /go/bin/* /bin/
EOF

Now, let's use this to build Go source from github.com/golang/example/hello:

$ docker build --build-arg gosrc=github.com/golang/example/hello GoBuild 
Sending build context to Docker daemon 2.048 kB
Step 1/4 : FROM golang:1.7
 ---> 7afbc2b03b9e
Step 2/4 : ARG gosrc
 ---> Using cache
 ---> 450e2a02c5dc
Step 3/4 : RUN go get ${gosrc}
 ---> Running in 1c5e27641489
 ---> d0c93c563988
Removing intermediate container 1c5e27641489
Step 4/4 : RUN cp /go/bin/* /bin/
 ---> Running in 28e989b5a531
 ---> b15dcb5216a1
Removing intermediate container 28e989b5a531
Successfully built b15dcb5216a1

If we inspect the image that was created, we note that it is a whopping 678 MB (the golang:1.7 base itself was 675 MB), and we can also see that is made up of a number of layers:

$ docker inspect b15dcb5216a1
[
    {
        ...
        "Size": 678257864,
        ...
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c",
                "sha256:0eb22bfb707db44a8e5ba46a21b2ac59c83dfa946228f04be511aba313bdc090",
                "sha256:30339f20ced009fc394410ac3360f387351641ed40d6b2a44b0d39098e2e2c40",
                "sha256:f4d2be23d5960387c35a14915ae0b29ddd2e9e275926a2129a9049f2231c0fc2",
                "sha256:d23a95ba38e5cbe6381573f82eb42921f92e8e9f3051903159b0cf96b316fe97",
                "sha256:677062ced7d30a66c16abaa9dcf3f8a06712d1bf9c966fc8eb8ab1948a4ee915",
                "sha256:8eb1c995f8b9b221059142cd3f6891e76a5fc6cbe332511ec3f7d0883503ff5a",
                "sha256:525ed957b386fd33667298230abf46ae2ccfed32679687fa62096b4f7c319531",
                "sha256:1aaf28531ad1742a2b0ef8147d08798b08950f27e7d3765b20e271e220285676"
            ]
        }
    }
]

The first part of this patch gives the ability to get some debug information about these layers. For example, here we pass the SHA256 diff ID (as returned under RootFS in docker inspect <image> output) to docker inspect for the final layer produced above:

$ docker inspect 1aaf28531ad1742a2b0ef8147d08798b08950f27e7d3765b20e271e220285676
[
    {
        "Name": "sha256:1aaf28531ad1742a2b0ef8147d08798b08950f27e7d3765b20e271e220285676",
        "Diffs": [
            {
                "Path": "bin/",
                "Sum256": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="
            },
            {
                "Path": "bin/hello",
                "Sum256": "xmOSG/IRLX3TsXH7SsgQ6j8mBz5nKcQMHszPvd9m2H4="
            }
        ]
    }
]

Now that we have our GoBuild/Dockerfile that can produce Go binaries for a given repo, now we can use this from within another Dockerfile to cherry-pick the output produced by the last command (e.g. the final layer).

For example:

$ cat <<'EOF' > Dockerfile
FROM alpine:latest
CHERRYPICK ["GoBuild/Dockerfile", "gosrc", "github.com/golang/example/hello"]
ENTRYPOINT ["/bin/hello"]
EOF

Introduces a CHERRYPICK instruction, which kicks off another image build (first argument is the Dockerfile to use, followed by optional build argument pairs). After the other image is built, it then cherry-picks the final layer to apply to its own image.

# docker build .
Sending build context to Docker daemon 4.608 kB
Step 1/3 : FROM alpine:latest
 ---> 88e169ea8f46
Step 2/3 : CHERRYPICK GoBuild/Dockerfile gosrc github.com/golang/example/hello
    Step 1/4 : FROM golang:1.7
     ---> 7afbc2b03b9e
    Step 2/4 : ARG gosrc
     ---> Using cache
     ---> 450e2a02c5dc
    Step 3/4 : RUN go get ${gosrc}
     ---> Using cache
     ---> d0c93c563988
    Step 4/4 : RUN cp /go/bin/* /bin/
     ---> Using cache
     ---> b15dcb5216a1
    Successfully built b15dcb5216a1
 ---> 42c01fe6d8fa
Removing intermediate container 8ea3ed5a79ac
Step 3/3 : ENTRYPOINT /bin/hello
 ---> Running in b7a0bc640ec5
 ---> a69b37e7459c
Removing intermediate container b7a0bc640ec5
Successfully built a69b37e7459c

Note it happily used the cache to rebuild b15dcb5216a1. If we inspect our new image we see:

$ docker inspect a69b37e7459c
[
    {
        ...
        "Size": 5621497,
        ...
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:60ab55d3379d47c1ba6b6225d59d10e1f52096ee9d5c816e42c635ccc57a5a2b",
                "sha256:1aaf28531ad1742a2b0ef8147d08798b08950f27e7d3765b20e271e220285676"
            ]
        }
    }
]

Total size 5.6 MB and we note that the second layer has the same diff ID as the one we created earlier.

To prove it isn't all smoke and mirrors, we execute an container using the image:

$ docker run --rm a69b37e7459c
Hello, Go examples!

The branch I created is here:
continusec/docker@master...continusec:addlayerinspectandcherrypick

Appreciate your thoughts on what I've done so far - and whether aspects are worth pursuing further. If there's interest I'd be happy to work on adding some tests etc to it to get it into better state for a pull request.

[davidawad]

The solution by @rainbreak worked for me who just wanted to clone a repo to a folder. In my example it was a wrapper for hubot, but it still worked just fine.

ADD https://api.github.com/repos/github/hubot/compare/master...HEAD /dev/null
RUN git clone https://github.com/github/hubot /hubot

WORKDIR /hubot

Thanks!

[thaJeztah]

I'm closing this issue because I don't see us implementing this (for reasons mentioned by @unclejack in #14704 (comment)).

As mentioned above there are some ways to work around this, also, docker 17.05 introduces multi-stage builds, which allow you to perform tasks (such as cloning a git repository) in a build-step, and use the results in a following build step, which will lead to smaller images, and allow more fine-grained use of the repository that was cloned.

[aeijdenberg]

The multi-stage stuff looks really good. Look forward to using it, thanks.

[mitar]

So on Docker Hub, if you are doing automatic builds from a private repository, you can add a deploy key to the user and then Docker Hub can fetches git submodules to other private repositories as well (if that user has access to them).

But that deploy key is not available during image building, so commands inside the Docker Hub image building cannot access private git repositories. Passing private keys in is messy, but having ADD access a private git URL, and reuse deploy keys from caller would be a great way to achieve this.

[rzr]

IHMO, this feature is still desirable

For the record, an alternate option to save bandwidth for github projects is to pull archive and unpack in container:

ADD https://github.com/$name/$project/archive/$branch.tar.gz /usr/local/src/$project-$branch.tar.gz
WORKDIR /usr/local/src/
RUN tar xvfz ${project}-$branch.tar.gz 
 # (...)

HTH

[codespearhead]

Here's a minimal working example of @rzr 's answer:

FROM alpine
WORKDIR /app
ADD https://github.com/Paguiar735/homework/archive/refs/heads/main.tar.gz main.tar.gz
RUN tar --strip-components=1 -xvf main.tar.gz
RUN rm -rf main.tar.gz

ENTRYPOINT ["ls"]

if you want to try it out on your end, add the above code snippet in a file named "Dockerfile" (without quotes), and run the following commands in the same folder as the Dockerfile.

docker build -t test .
docker run --rm test

The output will be the name of two files, which are all the current files there are in that repository.

[thaJeztah]

This has been implemented in the (currently the "labs", not yet in the "stable") Dockerfile front-end in moby/buildkit#2799

If you have BuildKit enabled (DOCKER_BUILDKIT=1), you can specify the docker/dockerfile:1.5-labs syntax to use that feature (at some point it will be included in the stable syntax);

# syntax=docker/dockerfile:1.5-labs
FROM alpine
ADD --keep-git-dir=true https://github.com/moby/buildkit.git#v0.10.1 /buildkit

DOCKER_BUILDKIT=1 docker build -t foo .
[+] Building 6.9s (11/11) FINISHED
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 169B                                                                                                                                                                           0.0s
 => resolve image config for docker.io/docker/dockerfile:1.5-labs                                                                                                                                              2.3s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                               0.0s
 => docker-image://docker.io/docker/dockerfile:1.5-labs@sha256:033d4afc7bdc581d28bc98b35598fffb1ead5a0dbe9ef2e00546e559b650e8da                                                                                0.1s
 => => resolve docker.io/docker/dockerfile:1.5-labs@sha256:033d4afc7bdc581d28bc98b35598fffb1ead5a0dbe9ef2e00546e559b650e8da                                                                                    0.0s
 => => sha256:e7f1764f75cb4ccd80dddbe45085e6cd8960d230785f08f3e1a1c744d18da4ff 2.10MB / 10.82MB                                                                                                                2.5s
 => => extracting sha256:e7f1764f75cb4ccd80dddbe45085e6cd8960d230785f08f3e1a1c744d18da4ff                                                                                                                      0.1s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                               1.2s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                                                                                  0.0s
 => git://github.com/moby/buildkit.git#v0.10.1                                                                                                                                                                 0.6s
 => [1/2] FROM docker.io/library/alpine@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a                                                                                                0.0s
 => => resolve docker.io/library/alpine@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a                                                                                                0.0s
 => CACHED [2/2] ADD --keep-git-dir=true https://github.com/moby/buildkit.git#v0.10.1 /buildkit                                                                                                                0.0s
 => exporting to image                                                                                                                                                                                         0.0s
 => => exporting layers                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:fd0d80903d17464e7d70a450cc8509a2d0069dd6e44b423d6a276a47123f6633                                                                                                              0.0s
 => => exporting config sha256:9d4c8d65f4eeddfaeda6126fd0a2e5ed01ad9d0e56ab0821ca05bb1f39eb9fd0                                                                                                                0.0s
 => => naming to docker.io/library/foo:latest                                                                                                                                                                  0.0s
 => => unpacking to docker.io/library/foo:latest                                                                                                                                                               0.0s

docker run --rm foo sh -c 'ls -la /buildkit'
total 408
drwxr-xr-x   24 root     root          4096 Jan 30 11:18 .
drwxr-xr-x    1 root     root          4096 Jan 30 11:27 ..
-rw-r--r--    1 root     root            16 Jan 30 11:18 .dockerignore
drwxr-xr-x    8 root     root          4096 Jan 30 11:18 .git
drwxr-xr-x    3 root     root          4096 Jan 30 11:18 .github
-rw-r--r--    1 root     root            37 Jan 30 11:18 .gitignore
-rw-r--r--    1 root     root           733 Jan 30 11:18 .golangci.yml
-rw-r--r--    1 root     root          1600 Jan 30 11:18 .mailmap
...