New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using docker-machine to import hosts using --driver=generic and/or --url, does not work. #1221

Open
smakam opened this Issue May 23, 2015 · 62 comments

Comments

Projects
None yet
@smakam

smakam commented May 23, 2015

hi folks
This is not an issue. I am sorry if this is the wrong place to ask this question. If this is not the right place, please point me to the right place.

I understand that we can use docker-machine to connect to different hosts using drivers like Virtualbox, cloud provider driver etc. If I already have a host running Docker inside baremetal Linux, how do we integrate this with docker-machine? Without docker-machine, I could do the same by running a Docker daemon on a particular port and connecting externally from docker client to the docker daemon ip and port. There is an option in docker-machine to create a host without any driver, is that for this purpose? I couldnt find how to use it to connect.

Thanks
Sreenivas

@nathanleclaire

This comment has been minimized.

Contributor

nathanleclaire commented May 23, 2015

Hi @smakam, I believe what you are looking for is the url driver: http://docs.docker.com/machine/#adding-a-host-without-a-driver

@smakam

This comment has been minimized.

smakam commented May 24, 2015

Hi @nathanleclaire
Thanks for the response.
I did look at the link you mentioned and I tried the following, no luck.

First, I tried without any TLS:
On Ubuntu machine I did this to start docker agent:
sudo docker -d -H unix:///var/run/docker.sock -H tcp://192.168.56.101:2376 &

On Windows, where I had docker-machine installed, I did this:
$ docker-machine create --url=tcp://192.168.56.101:2376 custom6
←[34mINFO←[0m[0000] "custom6" has been created and is now the active machine.
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
custom6 * none tcp://192.168.56.101:2376

I get this error:
$ docker-machine env custom6
←[31mFATA←[0m[0000] open C:\Users\srmakam.docker\machine\machines\custom6\ca.pem: The system cannot find the file specified.

Not sure if docker-machine enforces TLS. I tried starting docker agent with certificate and key and tried with docker client, no luck with that either.

Thanks
Sreenivas

@nathanleclaire

This comment has been minimized.

Contributor

nathanleclaire commented May 26, 2015

We do essentially mandate TLS in our current form. If you set up your own CA and certs/keys, I think you could probably use them via the --tls-ca-cert, --tls-ca-key gloabl options to Docker Machine. @ehazlett Any comment?

@smakam

This comment has been minimized.

smakam commented May 27, 2015

Hi

I tried with TLS, I still couldnt get it working, not sure what I am missing:
I started docker on my Ubuntu host:
sudo /usr/bin/docker -d --tlsverify --tlscacert=

/ca.pem --tlskey=/key.pem --tlscert=/cert.pem --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2376

Then I tried to create a docker-machine host without driver:
docker-machine --tls-client-cert=

/cert.pem --tls-ca-cert=/ca.pem --tls-client-key=/key.pem create --url=tcp://:2376 custom3

When I tried to see environment, I get the following error:
$ docker-machine env custom3
←[31mFATA←[0m[0000] open C:\Users\srmakam.docker\machine\machines\custom3\ca.pem: The system cannot find the file specified.

Thanks
Sreenivas

@ehazlett

This comment has been minimized.

Member

ehazlett commented May 27, 2015

You would most likely want to use existing CA, client, etc for all machine related stuff as the --tls-ca-cert etc settings are global.

As for your docker daemon, you want to use the CA and server cert /keys -- in the above you are using the CA cert, but client key and cert. You would need something like:

docker -d --tlsverify --tlscacert ca.pem --tlscakey ca-key.pem --tlscert server.pem --tlskey server-key.pem
@smakam

This comment has been minimized.

smakam commented May 27, 2015

@ehazlett
I tried to use global settings in "certs" directory for docker-machine create specifying options in commandline, still it complains about "ca.pem" missing in machines/\ directory.
How do I get "server.pem" and "server-key.pem"? Should I generate it?

I was able to connect between docker client and docker agent on separate machine using TLS without using docker-machine.

Thanks
Sreenivas

@ehazlett

This comment has been minimized.

Member

ehazlett commented May 28, 2015

Can you show the command line args used? If you specify the certs machine should just use them. If not, it's a bug :)

@smakam

This comment has been minimized.

smakam commented May 31, 2015

Hi @ehazlett
First, I started docker agent like this:
sudo /usr/bin/docker -d --tlsverify --tlscacert=/home/xxx/.docker/machine/certs/ca.pem --tlskey=/home/xxx/.docker/machine/certs/key.pem --tlscert=/home/xxx/.docker/machine/certs/cert.pem --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2376

Then I started docker-machine client like this:
docker-machine --tls-ca-cert=/home/xxx/.docker/machine/certs/ca.pem --tls-client-key=/home/xxx/.docker/machine/certs/key.pem --tls-client-cert=/home/xxx/.docker/machine/certs/cert.pem create --url=tcp://0.0.0.0:2376 custom3

I got this error when setting environment:
xxx@ubuntu:~$ docker-machine env custom3
open /home/xxx/.docker/machine/machines/custom3/ca.pem: no such file or directory

Here, I am running docker agent and docker-machine on same Ubuntu machine. I get similar error when running docker agent in Ubuntu and docker-machine in Windows.

Thanks
Sreenivas

@ehazlett

This comment has been minimized.

Member

ehazlett commented Jun 2, 2015

You should not be using key.pem and cert.pem for the Docker engine. The engine needs a server key/cert pair (Machine will create these).

For the environment, how did you create the custom3 machine? It looks like something went wrong during creation if that file doesn't exist.

@smakam

This comment has been minimized.

smakam commented Jun 2, 2015

Hi @ehazlett
How should I start docker engine? As per my understanding, docker-machine mandates using TLS.

This is how I created custom3 machine. This didnt give me any error.
docker-machine --tls-ca-cert=/home/xxx/.docker/machine/certs/ca.pem --tls-client-key=/home/xxx/.docker/machine/certs/key.pem --tls-client-cert=/home/xxx/.docker/machine/certs/cert.pem create --url=tcp://0.0.0.0:2376 custom3

Should I be creating server key/cert pair manually like what machine would do if driver was used with docker-machine?

Thanks
Sreenivas

@ehazlett

This comment has been minimized.

Member

ehazlett commented Jun 4, 2015

@smakam this should be correct. It should generate a server key based on that existing CA. I will do some testing to see if there is an issue.

@nathanleclaire

This comment has been minimized.

Contributor

nathanleclaire commented Jul 6, 2015

@smakam Any update on this issue or can we close it?

@smakam

This comment has been minimized.

smakam commented Jul 7, 2015

@nathanleclaire I am still not able to get it to work. I even tried with docker-machine 3.0 with generic driver procedure mentioned here(http://blog.docker.com/2015/06/docker-machine-0-3-0-deep-dive/)
This is what is mentioned:
docker-machine create -d generic
--generic-ssh-user ubuntu
--generic-ssh-key ~/Downloads/manually_created_key.pub
--generic-ip-address 12.34.56.78
jungle

I assume .pub is a typo and we need to give private key. I got 2 kinds of errors in 2 different hosts:

case 1:
Importing SSH key...
Error creating machine: exit status 1
You will want to check the provider to make sure the machine and associated resources were properly removed.

case 2:
Importing SSH key...
Error getting SSH command to check if the daemon is up: exit status 1
Error getting SSH command to check if the daemon is up: exit status 1

btw, where does docker-machine detailed logs get stored?
I tried with both docker-machine for windows and linux.

Thanks
Sreenivas

@yeasy

This comment has been minimized.

yeasy commented Jul 16, 2015

The same problem.
Both --url and -d generic ways do not work.

@jk-shah

This comment has been minimized.

jk-shah commented Jul 17, 2015

I tried the same steps as original poster and got similar issue at which point I tried the generic driver and ran into error similar to case 1 (CentOS 7)

@nunofgs

This comment has been minimized.

nunofgs commented Jul 29, 2015

Same problem. Can't use --url and specify the certificate.

@marktyers

This comment has been minimized.

marktyers commented Aug 5, 2015

Might I suggest the Docker team write a short tutorial taking us through step by step, it would be very helpful.

@zhanglongyang

This comment has been minimized.

zhanglongyang commented Aug 6, 2015

Same issue here with anyconnect vpn connected, well after I restarted my laptop without anyconnect, it is gone.

@narqo

This comment has been minimized.

Contributor

narqo commented Aug 14, 2015

+1 for the issue.
I've tried to use docker-machine create --url= with my existing docker setup on the remote host, but no luck. DM fails trying to get TLS from $HOME/.docker/machine/machines/<name>/ca.pem.

@miracle-in-sunday

This comment has been minimized.

miracle-in-sunday commented Aug 17, 2015

Same problem. I tried same steps, but still no server cert/keys generated.
docker-machine version 0.4.0
Docker version 1.8.0

@csokun

This comment has been minimized.

csokun commented Aug 18, 2015

Same problem. I tried same steps, but still no server cert/keys generated.
docker-machine version 0.4.1
Docker version 1.8.1

@nathanleclaire nathanleclaire changed the title from Using docker-machine to connect to hosts already running docker daemon/agent? to Using docker-machine to import hosts using --driver=generic and/or --url, does not work. Aug 18, 2015

@nathanleclaire

This comment has been minimized.

Contributor

nathanleclaire commented Aug 18, 2015

@csokun @miracle-in-sunday @narqo Generally it is assumed that using --url you "bring your own certs" although I do admit, that bit of the code has been languishing so it may be broken.

If you would like certficates and keys to be generated automatically, try the generic driver: https://docs.docker.com/machine/drivers/generic/

If that doesn't work for your use case, can I please ask that you file a separate issue detailing the exact steps you are taking, and the results you are seeing?

Thanks!

@bartslaman

This comment has been minimized.

bartslaman commented Aug 26, 2015

Hi Guys, Did you try with --virtualbox-hostonly-cidr speciefied? Worked for me:

BartSlaman@VLRNB176 ~
$ docker-machine create -d virtualbox --virtualbox-hostonly-cidr "192.168.99.100/24" dev4
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
To see how to connect Docker to this machine, run: C:\Program Files (x86)\Git\bin\docker-machine env dev4

BartSlaman@VLRNB176 ~
$ docker-machine env dev4
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.101:2376"
export DOCKER_CERT_PATH="C:\Users\BartSlaman.docker\machine\machines\dev4"
export DOCKER_MACHINE_NAME="dev4"

Regards
Bart Slaman

@soulski

This comment has been minimized.

soulski commented Sep 7, 2015

Any update on this issue? I tried --url with --tls-* but I got err "open /Users/user/.docker/machine/machine/ss/ca.pem no such file or directory". I use docker-machine version 0.4.1

@marlonpp

This comment has been minimized.

marlonpp commented Sep 14, 2015

Same here, but different error on creating:

Importing SSH key...
SSH cmd error!
command: sudo hostname internal && echo "internal" | sudo tee /etc/hostname
err : exit status 1
output : sudo: no tty present and no askpass program specified

And then when runing eval "$(docker-machine env internal)":
open /Users/marlon/.docker/machine/machines/internal/ca.pem: no such file or directory

That means the certs are not being generated.

Funny thing, I can ssh to the machine running "docker-machine ssh internal".

@rxacevedo

This comment has been minimized.

rxacevedo commented Sep 29, 2015

I was able to get --url to work and point to an existing Docker engine on DigitalOcean created with Docker Machine.

To use --url or the none driver, I just copied the existing ~/.docker/machine/machines/dobox/ folder on the machine that I used to create the host and deleted cert.pem, key.pem, id_rsa.pub, id_rsa and config.json (leaving ca.pem, server.pem and server-key.pem). I then generated a new client cert/key pair (https://docs.docker.com/articles/https/, starting at "For client authentication") inside the newly created directory from before and copied it over to the machine I was trying to connect from. Lastly, I added the remote host using docker-machine create --url=tcp://SOME_IP:2376 dobox and moved the certs to where Docker Machine expects them to be: ~/.docker/machine/machines/dobox/. The folder should be there already and contain config.json, so you are just adding your certs. I am not trying to change the TLS flags/auth scheme that Docker is using, which are:

--tlsverify \
--tlscacert="/home/roberto/.docker/machine/machines/dobox/ca.pem" \
--tlscert="/home/roberto/.docker/machine/machines/dobox/cert.pem" \
--tlskey="/home/roberto/.docker/machine/machines/dobox/key.pem" \
-H=tcp://SOME_IP:2376

Now I can docker $(docker-machine config dobox) images or just eval "$(docker-machine env dobox)" etc on the second client machine.

docker-machine version 0.4.0
Docker version 1.8.2

@kamaljeetrathi

This comment has been minimized.

kamaljeetrathi commented Oct 7, 2015

How to authenticate client and server in docker.whose username and password I have to configure with it
I am using https://docs.docker.com/reference/api/docker_remote_api_v1.20/
and have docker client and server on same host machine.
And also let me know if both are not on same machine.

@baio

This comment has been minimized.

baio commented Oct 21, 2015

Guys for the sake of the God, add tutorial how to import existent docker machines.
I've created 2 different cloud azure machines from 2 separates PCs and now I can't connect from PC to azure machine which were created on another.
Already spend 2 evenings, trying all suggestions and no result.

@dmp42

This comment has been minimized.

Member

dmp42 commented Oct 21, 2015

@jeanlaurent

This comment has been minimized.

Member

jeanlaurent commented Nov 30, 2015

The "no driver" is more a integration test driver used for development. But if you hack enough your setup can be used to register an existing machine. It is scheduled to be removed in a future release ( #2437 ). The current documentation is misleading on this none driver.

The generic driver should be used to register and properly install docker on any host.
Note that it will at the very least restart the docker daemon on the target host. But it's a reliable way to register any 'generic' machine into docker machine as long as you provide an ssh access.

@metasim

This comment has been minimized.

metasim commented Nov 30, 2015

@jeanlaurent the problem with the generic driver is that it mucks with your machine if docker is already installed (e.g. changing the hostname; running yum update). I think what many of us are trying to say is that the none driver has a first-class use case, and if it properly worked would be a valuable capability to many of us (in addition to the targeted drivers).

@vpusher

This comment has been minimized.

vpusher commented Nov 30, 2015

Agree with @metasim on this. The none driver is a so common use case. This is one of the most basic way to interact with remote docker API. It would be unfortunate to name it test. Peraphs the misleading comes from the fact that we don't want to create/manage a new machine (docker-machine create) but only connect to an existing machine ...

@jeanlaurent

This comment has been minimized.

Member

jeanlaurent commented Nov 30, 2015

I'm not saying we shouldn't be able to 'register' existing machine into docker-machine one way or the other, we just need to do it properly. There is a lot of confusion around the none driver, it doesn't work in a lot of places, and people are misusing it. The intent here is to prepare ground for a real register existing machine feature.

But once a machine is 'hacked' into docker-machine through the none driver there is a lot of stuff that is going to break down the chain -> restart, upgrade, and ssh obviously. Doing a manual change to the json file is as efficient as using the none driver to register a machine actually.

@metasim Upgrading the generic driver to not 'mucks with your machine` is one way of handling it.

@vpusher We may need to design a proper register feature.

@metasim

This comment has been minimized.

metasim commented Nov 30, 2015

@jeanlaurent Thanks for your response. I can see that that upgrading generic to handle this use case (including existing keys) may be a good way to go. However, in the spirit of "separation of concerns", I'd recommend considering a new driver (existing?, preconfig?, manual?, diy? :-) ) to handle this use case, particularly so that stability within this narrow context can be better established.

FYI. #2260 and #2269 (closed but not really addressed) were an attempt at capturing some of this confusion around the none driver

PS: I think it's wholly acceptable for some commands to be explicitly unsupported in this use case (e.g. restart, upgrade, etc.).

@jeanlaurent

This comment has been minimized.

Member

jeanlaurent commented Nov 30, 2015

@metasim Yup a dedicated driver is probably the best way actually to handle a register feature but not the only one, a dedicated command is another. But before deciding how to do it, we need to clarify/define in which case we all need to register a new machine without docker-machine installing docker on it.

If you take a look at PR #2442 for instance we ping the docker host for the docker version. Because we want to be able to provide upgrading at some point, or warn you that your docker host is too old with your current docker client. With a machine on which we didn't install the docker daemon this will prove difficult or at the very least very flaky.

If we consider that we can update the generic driver in some way, I'm wondering for which use case someone would like to register a machine into docker-machine without providing ssh access ?

What's yours ? Let's list them.

@nathanleclaire

This comment has been minimized.

Contributor

nathanleclaire commented Nov 30, 2015

Some commentary on defining the use case for none and/or potentially deprecating it here: #2437 (comment)

My feeling is that we (the Machine team) should:

  • Follow Docker's existing guidelines about deprecation of features (deprecate over 2 release cycles).
  • Consequently, if we're going to deprecate the none driver, we should wait to completely remove it until 0.7.0.
  • In the interim, fix the existing show-stopper bugs with the none driver, and work towards defining the ideal future workflow (whether it's import/register or changes to the generic driver) so that interested users can follow along and have adequate time to prepare for changes coming down the pipe.

@dgageot @jeanlaurent How does that sound?

@metasim

This comment has been minimized.

metasim commented Dec 1, 2015

@jeanlaurent I can see that ssh would pretty much be required, which isn't so bad. Up to now I'd just assumed most everything could be done over the REST API.

@gordontyler

This comment has been minimized.

gordontyler commented Dec 1, 2015

I use docker-machine mostly as a tool to setup the env variables required to make docker talk to another host. I never use the upgrade/restart etc. commands. My Docker hosts are in vSphere with a custom template created through a vOrchestrator workflow -- using the vSphere driver in docker-machine is not an option.

That said, I realize that my use-case is not the only use-case, but I hope that it's something to consider at least.

@gordontyler

This comment has been minimized.

gordontyler commented Dec 1, 2015

I could probably replace docker-machine with a shell alias, for my purposes at least.

@bonitao

This comment has been minimized.

bonitao commented Dec 1, 2015

Here is a sample use case in case it helps:

I would like to create my ec2 machines in amazon using elasticbeanstalk instead of docker-machine, because elasticbeanstalk has a lot of goodies (like auto-scaling and machine restarts). I would like to register those with docker-machine and control them using docker-swarm.

Upgrading those machines to have docker 1.9 is trivial, but they run some amazon linux (ancient centos fork) and generic does not work with them. Because the docker engine there does not have tls, the trick I use to manage them remotely with plain docker is to expose the docker.sock in localhost:2375 using socat (much like in https://github.com/sequenceiq/docker-socat) and do a ssh tunnel from my local machine with something like ssh -i id_rsa ubuntu@ec2.xxx -L 2375:localhost:2375 -N . Then I can connect over the network with docker --tls=false -H tcp://localhost:2375.

It is a lot of gymnastics. The combination of ssh access and a working docker setup (even without ssl) should be all that docker-machine needs for many interesting cases (no upgrade, restart as other noticed, but I never use those anyway).

Hope that is useful.

@metasim

This comment has been minimized.

metasim commented Dec 2, 2015

@bonitao Glad you spelled that out. I also have the "tunnel through ssh" use case in some enterprise engagements.

@marcellodesales

This comment has been minimized.

marcellodesales commented Dec 18, 2015

I'm just starting trying again to use Docker machine with existing hosts... I agree with @metasim statements on around the none driver... It has value today for the community, since the generic is not yet doing what its intention is... I spent inumerous hours trying to get the none working to learn that there's a proposal to get it renamed to test. @nathanleclaire Any plans to make the generic driver work as described about???

I speng a good amount of time on #2628 and this is a use case that we have... Supporting existing teams across the company with docker...

@discoverydev

This comment has been minimized.

discoverydev commented Dec 22, 2015

I've been googling, assuming I was just completely missing some understanding, trying to figure out how to connect up to an azure docker machine from my CI server that I created elsewhere. I ended up here and am surprised this actually is not possible (without hackery).

I created and got azure docker machine instances fully up and running from a workstation and I just want to be able to control it and deploy to it from CI scripts, which can run from a number of CI slave instances. Is there really no official way this is supposed to be accomplished?

@andrecp

This comment has been minimized.

andrecp commented Jan 30, 2016

I've updated my docker and now I can no longer connect to my old docker-machine instance. I get

Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error validating certificates for host "xxxxx:2376": open : no such file or directory
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

the regeenrate-certs commands gives me

Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Detecting the provisioner...
Installing Docker...
Copying certs to the local machine directory...
Copying ca.pem to machine dir failed: open : no such file or directory

funny thing is that I can

docker-machine ssh

without any problems..

any fixes?

@devcrust

This comment has been minimized.

devcrust commented Feb 1, 2016

Scenario: Existing and working server with Docker and TLS enabled.

Add existing server/machine to docker-machine:

docker-machine --tls-ca-cert path/to/ca.pem --tls-ca-key path/to/ca-key.pem --tls-client-cert path/to/client.pem --tls-client-key path/to/client-key.pem create --driver none --url tcp://HOST:2376 NAME

In your user directory (~/.docker/machine/machines/NAME) add the same client certificate as "cert.pem" and "server.pem" and the client certificate key as "key.pem" and "server-key.pem" also adjust your config.json to include the relevant SSH settings...

@smakam

This comment has been minimized.

smakam commented Mar 23, 2016

I had raised this issue. I recently got this working and I have put the instructions here(https://sreeninet.wordpress.com/2015/05/31/docker-machine/) in case anyone wants to refer.

@metasim

This comment has been minimized.

metasim commented Apr 20, 2016

Here's a script that's kinda working for me:
#3344 (comment)

@l15k4

This comment has been minimized.

l15k4 commented May 6, 2016

@devcrust where exactly are those :

path/to/ca.pem
path/to/ca-key.pem
path/to/client.pem
path/to/client-key.pem

???

$ ls certs/
ca-key.pem  ca.pem  cert.pem  key.pem
$ ls machines/adhoc/
ca.pem  cert.pem  config.json  id_rsa  id_rsa.pub  key.pem  server-key.pem  server.pem

I cannot set it up properly. I have this issue too :

Copying ca.pem to machine dir failed: open : no such file or directory

after upgrading to 0.6.0 .... I lost tcp access to 12 machines

@pcgeek86

This comment has been minimized.

pcgeek86 commented Jul 4, 2016

@nathanleclaire Any update on this? I'm trying to figure out how I'd connect to a Docker Host running in Microsoft Azure, that I created from a different computer, using docker-machine. Right now, I have zero solutions.

@myuseringithub

This comment has been minimized.

myuseringithub commented Feb 9, 2017

Solution: (you wish)

docker-machine add <🖥️ name> --driver <☁️️ provider driver>

So sad that so much time passed and still no feature is introduced to solve this problem.

@ivandir

This comment has been minimized.

ivandir commented May 10, 2017

For what it's worth. You can create a machine using the generic driver but that will restart all your containers.
Adding restart: always on your containers will guarantee that they don't stop. I'd second the docker-machine add option

@drewlsvern

This comment has been minimized.

drewlsvern commented Oct 27, 2017

I'm also game for having a docker-machine add option.

@schmunk42

This comment has been minimized.

schmunk42 commented Oct 31, 2017

Here are two projects with different approaches to machine sharing:

@x-yuri

This comment has been minimized.

x-yuri commented Nov 16, 2018

@jeanlaurent

What's yours ? Let's list them.

My idea was to connect my local machine to an existing physical server with a load of containers running. Doing docker-machine create --driver generic will most likely stop them, and more. I wonder why it needs to restart docker...

Well, I can simply run commands over ssh, but from its description it looked like docker-machine could be used as well.

But then, if you created a VM from a computer, and want to manage it from another one. Or you reinstalled your local OS... Or want to delegate control over the VM to somebody else...

P.S. I'm making my first steps with Docker, so there might be points I'm missing...

@schmunk42

This comment has been minimized.

schmunk42 commented Nov 16, 2018

@x-yuri To "create" a machine manually, just copy its files from .docker/machine/machines and adjust its paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment