Add utility to create a build context from a git repository.

[apeace]

The reason is because it won't work if you pass a private git repository as the build URL.

See docker/compose#2856 for more information, including a simple test-case by @zkf.

Instead, docker-py should follow the same steps as docker: if the build URL is a git repository, it should clone the repo locally and pass as a tarball to the daemon.

I believe the problem is caused by how docker-py performs a build (here) instead of following the same steps as docker.

Please let me know if I'm misunderstanding the problem or if there's any other way I can help.

Here is my version information:

OS X El Capitan 10.11.2 (15C50)
docker-py==1.8.0.dev0
Python 2.7.9
Client:
 Version:      1.11.0-dev
 API version:  1.23
 Go version:   go1.6
 Git commit:   bc730f3-unsupported
 Built:        Tue Mar  8 19:01:29 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0-dev
 API version:  1.23
 Go version:   go1.6
 Git commit:   bc730f3-unsupported
 Built:        Tue Mar  8 19:01:29 2016
 OS/Arch:      linux/amd64

[shin-]

I think both options should exist. For public repositories, just passing the git URL is definitely faster.

Feel free to submit a patch to add that feature.

[apeace]

Thanks @shin-

I can work on a patch for this, though I'm sure it will take me some time :) I have a couple questions for you:

1. Are you sure this wouldn't be considered a bug? I believe (though have not yet tested) that this would also not work for public repositories if you use SSH syntax (git@github.com:user/repo.git). Essentially I think git SSH URLs will not work. Perhaps the bug should be clarified to say "git SSH URLs" instead of any "git URL"?
2. If indeed this is not a bug, could you give me some guidance on how a docker-py client might enable this feature? For example, maybe a client like docker-compose might pass a flag (e.g. to this call)? Any guidance on that would be appreciated--I will incorporate it into my patch.

[shin-]

Ah, I just tested it and it seems you are correct - public repos seem to fail too here. However, the engine definitely recognizes the syntax since it's trying to process it (Cloning into '/var/lib/docker/tmp/docker-build-git307830553'...) - so I would argue that this is actually a bug at the engine level. I'm certain it used to work before as well.
We should open an issue on the docker/docker issue tracker to report it and see if this can be fixed.

As far as 2 is concerned, the way I'd implement it is have a function in utils to clone a git repository (maybe this should be in its own package - it's actually nontrivial) and tar it into a file object (pointing to a temp file on the disk ideally). Then this can be passed to the Client.build method through the fileobj param, with custom_context set to True. If those utils exist compose can easily leverage them afterwards.

[apeace]

@shin- I'm not sure there is a bug at the engine level. If you read my last comment on docker/compose#2856 I think the problem is that the git clone is happening from inside the docker daemon. This means it doesn't have access to your SSH config.

One possible fix is moby/moby#6396, but I believe the docker binary has a simpler workaround that docker-py could use too.

The docker binary itself will perform the clone locally first (when you run docker build), before sending a build context to the docker daemon. What I'm suggesting is that docker-py should emulate this same behavior. Does that seem right to you?

Therefore I am thinking the implementation in docker-py might look like this:

• Create a util to clone a git repo and tar it like you said.
• In BuildApiMixin.build, IF path begins with git@, then always clone/tar the repo locally, and provide it do the docker daemon.

This way clients (like docker-compose) don't have to change, and they get a fixed behavior that should always be breaking right now.

[shin-]

My daemon is on the same machine as my client, so it should have access to my SSH config just fine, but I'm still experiencing the issue.

[dnephin]

I think the daemon probably runs as a different user, so it won't be looking in the right directory for the ssh config?

[apeace]

I believe @dnephin is right, the daemon is most likely running as a different user. I think that is why docker has the behavior of cloning locally instead of passing the remote param to the daemon.

[shin-]

Ah, good point. After copying my .ssh folder into /root, I am able to build from a git-ssh URL.

In [1]: import docker

In [2]: c = docker.Client()

In [3]: strm = c.build('git@github.com:docker/docker-py.git', stream=True)

In [4]: for line in strm:
   ...:     print line
   ...:     
{"stream":"Step 1 : FROM python:2.7\n"}

{"stream":" ---\u003e 31093b2dabe2\n"}
[...]
{"stream":"Successfully built 16580dbca2de\n"}

[shin-]

Regardless, I still think the second part of #980 (comment) is the best approach.

[apeace]

@shin- sure. I should have time to work on a patch over the next couple weeks.

Just to be clear, you're saying clients should have the choice to either:

• Use docker.util.clone_repo (or something), and then pass with fileobj and custom_context
• OR, pass path=git@github... and expect it will be passed to the docker daemon

Thank you again! :)

[shin-]

Just to be clear, you're saying clients should have the choice to either:

• Use docker.util.clone_repo (or something), and then pass with fileobj and custom_context
• OR, pass path=git@github... and expect it will be passed to the docker daemon

Yep! Thanks for offering to take it on!

[somombo]

Hi @apeace I recently ran into issues noted in #2856. How is the progress going? Thanks so much for taking this on!