Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export / import machines #23

Open
bfirsh opened this issue Dec 6, 2014 · 67 comments
Open

Export / import machines #23

bfirsh opened this issue Dec 6, 2014 · 67 comments
Milestone

Comments

@bfirsh
Copy link
Contributor

@bfirsh bfirsh commented Dec 6, 2014

Something like this should work for transferring machines:

 $ machine export test | ssh anotherhost machine import

(Maybe we could use machine inspect?)

@bfirsh bfirsh changed the title Import machines Export / import machines Dec 6, 2014
@arminc
Copy link

@arminc arminc commented Dec 6, 2014

What's the gain on this one? The goal of using machine from my perspective is to set up a simple machine with docker on it (no fancy stuff or custom things). With that in mind creating or export->importing a machine is exactly the same and does not give me any benefit.

@bfirsh
Copy link
Contributor Author

@bfirsh bfirsh commented Dec 6, 2014

When you create a machine, you can only use it and manage it from a single computer. You might want to:

  1. backup your hosts
  2. transfer them to another computer
  3. share them with a team member
@waitingkuo
Copy link

@waitingkuo waitingkuo commented Dec 6, 2014

Should we have a drop command so that we can remove the machine from the list without terminating it? I think it might be useful after some machines are exported and transferred to another computer.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Dec 6, 2014

What exactly will the saved file contain? i.e.; Will it contain the token for connecting to the daemon?

Ideally, there should be something to manage computers/people to have access to the host. Thinking along the lines of a one-time token that can be to connect a new computer to an existing host.

@cjellick
Copy link

@cjellick cjellick commented Jan 9, 2015

@bfirsh @waitingkuo This seems like a really useful feature that we'd use at Rancher (we'd like to use machine server-side).

Any reason the implementation (#29) seems to have stalled. Anything I could do to attempt to hep?

@gastonmorixe
Copy link

@gastonmorixe gastonmorixe commented Apr 24, 2015

+1 would be awesome

@Starefossen
Copy link

@Starefossen Starefossen commented May 6, 2015

+1

@urish
Copy link

@urish urish commented Jun 27, 2015

+1

1 similar comment
@kevin-coyle
Copy link

@kevin-coyle kevin-coyle commented Sep 8, 2015

+1

@kevinsimper
Copy link

@kevinsimper kevinsimper commented Sep 15, 2015

We have create a tool that we use ourselves, to make it easy to export docker-machines. It basicly exports all the certificates ssh keys. Feedback is welcome!

https://github.com/blackbeardapp/docker-machine-export

@uptownhr
Copy link

@uptownhr uptownhr commented Oct 13, 2015

might be an overkill but something like a docker-machine registry would be awesome!

@cusspvz
Copy link

@cusspvz cusspvz commented Oct 13, 2015

Currently we are using a git repo for sharing docker settings, problem is that some of us have virtualbox machines and thats the down-side

@jeanlaurent jeanlaurent added this to the 0.X.0 milestone Nov 3, 2015
@docteurklein
Copy link

@docteurklein docteurklein commented Nov 16, 2015

@kevinsimper how do you import them back?

@saada
Copy link

@saada saada commented Nov 20, 2015

Does this issue include the idea of sharing swarm clusters between developers?

@docteurklein
Copy link

@docteurklein docteurklein commented Nov 23, 2015

@saada I think it only covers credentials sharing, not cluster config.
It would be awesome tho to have this ability in machine.
Currently I only know of https://github.com/efrecon/machinery and https://github.com/nathanleclaire/moby

@micheletedeschi
Copy link

@micheletedeschi micheletedeschi commented Dec 2, 2015

+1

@tonysickpony
Copy link

@tonysickpony tonysickpony commented Dec 14, 2015

I am looking after the capability to share machine between users in the same hosts.
And is there a workaround for this ? I am copying the whole ~/.docker/machine/ folder for now

@fuse
Copy link

@fuse fuse commented Jan 16, 2016

+1

5 similar comments
@ffissore
Copy link

@ffissore ffissore commented Jan 25, 2016

+1

@shanewow
Copy link

@shanewow shanewow commented Feb 7, 2016

+1

@montytorr
Copy link

@montytorr montytorr commented Feb 9, 2016

+1

@toh2526
Copy link

@toh2526 toh2526 commented Feb 11, 2016

+1

@lukaville
Copy link

@lukaville lukaville commented Feb 17, 2016

+1

@opHASnoNAME
Copy link

@opHASnoNAME opHASnoNAME commented Jul 7, 2016

+1

@schmunk42
Copy link

@schmunk42 schmunk42 commented Sep 12, 2016

We're using a docker container for this task - what else 😄

The idea is also to have the same environment (eg. cert configuration paths) for all users, mounted as a host-volume. You can use docker-machine, docker-compose and docker in containers created from roj. And you can pass the data-folder as a zip via email or put it into a private git repo.

@ricardodantas
Copy link

@ricardodantas ricardodantas commented Oct 19, 2016

+1

@jcrben
Copy link

@jcrben jcrben commented Nov 13, 2016

I'm not sure my use-case is the same as others - I've created a couple different machines for different purposes and I want to back up their existence and (minor) configuration. I'd rather just have some simple text files which are used to recreate the boxes.

https://www.npmjs.com/package/machine-share doesn't seem to respect my MACHINE_STORAGE_PATH. schickling's script backs up couple large binary files (e.g., boot2docker.iso). https://www.npmjs.com/package/@mumbacloud/dmport threw a couple errors.

@montanaflynn
Copy link

@montanaflynn montanaflynn commented Nov 30, 2016

We also want this functionality, we've created a machine on gcloud using docker-machine for development and I want to allow my coworker to exec into the container and view the logs.

@bmmathe
Copy link

@bmmathe bmmathe commented Jan 13, 2017

I'm actually surprised this isn't supported already. My example is I created a docker swarm cluster in Azure from my laptop (without thinking this all the way through) and now I want other devs to be able to manage the cluster so I created a shared VM in Azure that everyone can share. I installed docker-machine and now I need a way to import the machines from my laptop to our jump box in Azure.

@colthreepv
Copy link

@colthreepv colthreepv commented Jan 13, 2017

@jcrben Hi, what about posting a full issue on machine-share, regarding the MACHINE_STORAGE_PATH?

I recently reviewed bhurlow's work, and I am curious to see if something breaks

@schmunk42
Copy link

@schmunk42 schmunk42 commented Jan 14, 2017

@jcrben Did you also try https://github.com/dmstr/docker-roj? ... I am just curios.

@ntwrkguru
Copy link

@ntwrkguru ntwrkguru commented May 1, 2017

People, for the love of all things proper, please 👍 the OP and not as a discrete comment. "+1" spam distracts from the conversation.

@lnshi
Copy link

@lnshi lnshi commented Sep 7, 2017

FYI: #3212

@pedrodevoto
Copy link

@pedrodevoto pedrodevoto commented Jan 2, 2018

@lnshi I think that the problem with importing an existing docker server with docker-machine create -d generic ..., is that it recreates certificates and restarts Docker, so not appropriate for a production/teamwork scenario.

@lnshi
Copy link

@lnshi lnshi commented Jan 2, 2018

@pedrodevoto I know, I just mentioned that issues, since ppl have discussed there a lot.
For now I don't think there is a proper solution yet.

@Fedcomp
Copy link

@Fedcomp Fedcomp commented Nov 3, 2018

@ntwrkguru still, it makes issue relevant and alive. There's no notification on emojis.

@ntwrkguru
Copy link

@ntwrkguru ntwrkguru commented Nov 3, 2018

This issue is nearly 4 years old, I doubt that a few more +1 comments will suddenly cause it to become a priority.

@exhuma
Copy link

@exhuma exhuma commented Nov 3, 2018

@ntwrkguru As long as the docker devs sort issues by 👍, it should be on their radar. It's the second-most upvoted issue at the moment. So keeping the 👍 coming will make sure that it stays on top.

Now, I don't know how the docker team prioritises issues, but I bet they also have paying customers, which I bet will always come first before looking at the public community feature requests.

@adilinden
Copy link

@adilinden adilinden commented Nov 8, 2018

The same: #1328 ?

I am just getting used to docker and built my first host using docker-machine. Now one day later, I am trying to connect to my docker host using a different workstation... Surprised to see that that's no possible, at least not without some major headaches. Seems like an essential feature to allow teams to manage hosts...

@BretFisher
Copy link

@BretFisher BretFisher commented Nov 8, 2018

That's because docker-machine was never designed to be a team tool. It's mean for an individual to easily create VM's that auto-install docker and setup dockerd TLS certs for remote cli management.

If you're trying to build and deploy docker environments for teams, you likely want to use a different tool.

@adilinden
Copy link

@adilinden adilinden commented Nov 8, 2018

@BretFisher and what tools would that be? Not that I am wanting to use it for teams, just from 3 different workstations...

@BretFisher
Copy link

@BretFisher BretFisher commented Nov 8, 2018

One of the big benefits of docker-machine is that it sets up TLS on the docker engine for remote cli management.

Today Docker Engine 18.09 was released and with it a great new feature that allows remote cli management using SSH tunneling (of sorts) and is a much better approach IMO for those of us without the need for full of docker engine RBAC. For those using docker-machine simply to make remote management easier, I recommend checking this out in 18.09. Both your docker server and client need to be on 18.09, and as long as you can SSH in, you can use the cli remotely.

Docker Captain Luc Juggery has a quick write-up on this. I expect it'll become my default way to use docker remotely.

@ntwrkguru
Copy link

@ntwrkguru ntwrkguru commented Nov 8, 2018

I get what you're saying @BretFisher, but it would be nice to have it in docker-machine, since it keeps track of the remotes anyway. If i'm going to manage them over SSH, then I likely won't even bother using docker-machine to deploy them, and instead use Ansible. Docker-machine is yet another tool that could be great, but is relegated to mediocre due to lack of development.

@BretFisher
Copy link

@BretFisher BretFisher commented Nov 8, 2018

@adilinden for a single user options include:

  1. sync ~/.docker/machine/machines between those machines in a private and/or encrypted git repo. You may also need ~/.docker/machine/certs too, I'm not sure. It needs to be reasonably secure since it's storing private certs.

  2. Pick a AMI, Droplet, etc. that already has docker installed and deploy those. Then use the new 18.09 SSH feature to remotely control the docker engine.

  3. Try one of the many tools that export/import the machine data directories I listed above.

@adilinden
Copy link

@adilinden adilinden commented Nov 8, 2018

@BretFisher I really like #2, it seems the best option by far for my scenario as all the knobs and buttons are in place for that already. Switching hosts looks trivial as well, as it is simply setting a single environment variable.

Now just to figure out how to get Docker for Mac to 18.09 instead 18.06.1-ce-mac73 (26764).

@BretFisher
Copy link

@BretFisher BretFisher commented Nov 8, 2018

@ntwrkguru docker-machine is great for one-off machines or personal setups or someone who maybe only needs ~10 servers max, but if you've ever tried to manage 10+ servers with it, you'll see its limits right away. I wouldn't say that's due to lack of development, but rather "not within scope" of the tools original goals.

On the SSH, you wouldn't use ansible in this case, what I'm saying is once you've deployed any docker server (ansible or otherwise) that runs 18.09+, as long as you have SSH permission to that server, you can now do something like:

docker -H ssh://user@10.10.10.10 run -p 80:80 nginx
or
DOCKER_HOST=ssh://user@10.10.10.10 docker system prune

For controlling that engine remotely without any special config or setup. Being able to export an envvar and then run a bunch of commands against that engine (without wrapping them in ansible, etc.) is very handy.

@BretFisher
Copy link

@BretFisher BretFisher commented Nov 8, 2018

@adilinden the Linux release just dropped today, so you'll need to give the teams a few days or week to roll out all the downstream products that use it :). If you want it today, use the edge release which has the 18.09 beta and works fine for me on Mac.

@ntwrkguru
Copy link

@ntwrkguru ntwrkguru commented Nov 8, 2018

Admittedly, I haven't messed with docker-machine in several years. I have 4 servers left running plain Docker engine.

So, what's the benefit between docker -H ssh://user@10.10.10.10 run -p 80:80 nginx and ssh://user@10.10.10.10 docker run -p 80:80 nginx?

[edit] Not being argumentative; I'm wondering if there's a clear benefit.

@adilinden
Copy link

@adilinden adilinden commented Nov 9, 2018

@ntwrkguru I can build from a Docker file on my workstation using the docker -H ssh://user@10.10.10.10 . command. Whereas the same fails (as expected) using ssh -luser 10.10.10.10 docker build .

@adilinden
Copy link

@adilinden adilinden commented Nov 10, 2018

Just for information, can also use ssh for socket forwarding. Per https://medium.com/@dperny/forwarding-the-docker-socket-over-ssh-e6567cfab160, but with a slight modification.

In one terminal session run

ssh -nNT -L $(pwd)/docker.sock:/var/run/docker.sock -l user 10.10.10.10

In another run

docker -H "unix:///$(pwd)/docker.sock" run -p 80:80 nginx

or

export DOCKER_HOST="unix:///$(pwd)/docker.sock"
docker run -p 80:80 nginx
@usr42
Copy link

@usr42 usr42 commented Nov 23, 2018

I had the same issue and I created shell scripts to do the backup and the restore:
https://github.com/usr42/docker-machine-backup

Only the really needed data is backed up. For instance no iso files are inside of the backup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.