Proposal: Host management #8681

Closed
bfirsh opened this Issue Oct 21, 2014 · 31 comments

Comments

Projects
None yet
@bfirsh
Contributor

bfirsh commented Oct 21, 2014

Getting started with Docker is a pain. If you want to use Docker locally, you have to install boot2docker and figure out how to install and run it. If you want to use Docker remotely, you have to figure out how to spin up an instance on a cloud provider, install Docker, set up certificates to secure the connection, etc.

I am proposing we add a way of managing Docker hosts from within the client itself. It’ll let you create hosts on both cloud providers and local hypervisors, then set up the client to point at them.

It works a bit like this:

$ docker hosts
NAME         ACTIVE   DRIVER         STATE     URL
default      *                                 unix:///var/run/docker.sock

$ docker hosts create -d virtualbox dev
[info] Downloading boot2docker...
[info] Creating SSH key...
[info] Creating Virtualbox VM...
[info] Starting Virtualbox VM...
[info] Waiting for host to start...
[info] "dev" has been created and is now the active host. Docker commands will now run against that host.

$ docker hosts
NAME      ACTIVE   DRIVER         STATE     URL
default                                     unix:///var/run/docker.sock
dev       *        virtualbox     Running   tcp://192.168.99.100:2375

$ docker run busybox echo hello I am dev
Unable to find image 'busybox' locally
Pulling repository busybox
e72ac664f4f0: Download complete
511136ea3c5a: Download complete
df7546f9f060: Download complete
e433a6c5b276: Download complete
hello I am dev

$ docker hosts create -d digitalocean --digitalocean-access-token=... staging
[info] Creating SSH key...
[info] Creating Digital Ocean droplet...
[info] Waiting for SSH...
[info] "staging" has been created and is now the active host. Docker commands will now run against that host.

$ docker hosts
NAME      ACTIVE   DRIVER         STATE     URL
default                                     unix:///var/run/docker.sock
dev                virtualbox     Running   tcp://192.168.99.100:2375
staging   *        digitalocean   Running   tcp://104.131.184.151:2375

$ docker run busybox echo hello I am staging
Unable to find image 'busybox' locally
Pulling repository busybox
e72ac664f4f0: Download complete
511136ea3c5a: Download complete
df7546f9f060: Download complete
e433a6c5b276: Download complete
hello I am staging

$ docker -H dev busybox echo hello I am dev
hello I am dev

$ docker hosts active dev
NAME      ACTIVE   DRIVER         STATE     URL
default                                     unix:///var/run/docker.sock
dev       *        virtualbox     Running   tcp://192.168.99.100:2375
staging            digitalocean   Running   tcp://104.131.184.151:2375

Initially, it will ship with a Virtualbox driver to replace the boot2docker client, and various drivers for cloud providers.

Active host

A host can be “active”, which means Docker commands will run against that host. This is a bit like selecting a branch in Git.

Client/daemon authentication and encryption

TLS will be enabled by default in the same release this ships.

Backwards compatibility

Docker ships with a hardcoded “default” host. The default host is always available, cannot be removed, and cannot be changed. When it is active, Docker behaves as it does at the moment (reads DOCKER_HOST, etc).

The -H option can be used to override the active host. If the host specified with -H is not prefixed with a protocol, Docker will attempt to look up a host of that name and use it. If there is a protocol, Docker will treat it as a URL to connect to.

Storage of host configuration

Hosts will be stored in flat files in your home directory so it is simple and accessible.

The configuration for a host is stored in ~/.docker/hosts/<name>/config.json, and a driver can store arbitrary files in ~/.docker/hosts/<name>/ (SSH keys, disk images, etc).

A file called ~/.docker/hosts/.active stores the name of the current active host.

Questions

Do we pick a single operating system? Is it the driver’s responsibility to choose the “best” operating system for a provider? Do we offer a choice of operating system?

How do we upgrade Docker on the hosts? Is this the driver’s responsibility? Is this the operating system’s responsibility? Could it be the Docker daemon’s responsibility?

Builds

As of 36583edf5c25fff1d33db411cdc2fbaafa2e5ea9

See also

@jeffmendoza

This comment has been minimized.

Show comment
Hide comment
@jeffmendoza

jeffmendoza Oct 21, 2014

Ben, can you outline how the client's public key will get placed in the server's "authorized_keys" upon new host creation? I understand that we'll need to merge with the libtrust work before getting this fully implemented.

Ben, can you outline how the client's public key will get placed in the server's "authorized_keys" upon new host creation? I understand that we'll need to merge with the libtrust work before getting this fully implemented.

@proppy

This comment has been minimized.

Show comment
Hide comment
@proppy

proppy Oct 21, 2014

Contributor

@bfirsh can you detail the integration points for host drivers? Are those similar to boot2docker-cli drivers?

Also can you hint at the implication on the installation experience for new user? Does that mean the boot2docker installer morph into a docker client install with support drivers?

Contributor

proppy commented Oct 21, 2014

@bfirsh can you detail the integration points for host drivers? Are those similar to boot2docker-cli drivers?

Also can you hint at the implication on the installation experience for new user? Does that mean the boot2docker installer morph into a docker client install with support drivers?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Oct 21, 2014

Contributor

Driver's responsibility to select OS and if possible (which should be in most cases) user specify-able, as well as a bootstrapping script. If the driver's provider provides boot2docker as an image, it should be used as a default.

Contributor

cpuguy83 commented Oct 21, 2014

Driver's responsibility to select OS and if possible (which should be in most cases) user specify-able, as well as a bootstrapping script. If the driver's provider provides boot2docker as an image, it should be used as a default.

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Oct 22, 2014

Contributor

@proppy Right now drivers implement this interface:

// Driver defines how a host is created and controlled. Different types of
// driver represent different ways hosts can be created (e.g. different
// hypervisors, different cloud providers)
type Driver interface {
    DriverName() string
    GetURL() (string, error)
    GetIP() (string, error)
    GetState() (state.State, error)
    Create() error
    SetConfigFromFlags(flags interface{}) error
    Remove() error
    Start() error
    Stop() error
    Restart() error
    Kill() error
    GetSSHCommand(args ...string) *exec.Cmd
    // Pause() error
}

and register themselves with a _ import in api/client/commands.go.

Feedback

First off, awesome work @bfirsh , @cpuguy83 and countless others who have been involved in this idea and moving this implementation forward. It's very exciting stuff.

I got interested in the movement around hosts and went off and implemented an Amazon EC2 driver. It's a lot of fun to be able to bootstrap a host pretty quickly from the command line.

Ben and the boot2docker-cli maintainers (which is where the original driver code came from) did a great job implementing an interface that's pretty easy to write to. Like anything new, there's plenty of rough surfaces that need to get ironed out before we even consider merging this into Docker.

My feedback on the proposal is as follows:

Implement docker hosts inspect foo

I've made a PR for this (bfirsh#4), I think it is a vital part of making an implementation of this succeed. Users shouldn't have to know of ~/.docker/hosts/myhost/config.json as an implementation detail.

Provide a way to track (and probably to plan) infrastructure changes

What is Docker about? Docker is about repeatability and about speed. If I can't take what I've created with Docker and repeat it elsewhere, fast, it is not much use to me. Therefore if Docker is going to get into the business of creating hosts to run Docker, it must provide a way to do so which is tracked and repeatable. Something like git is not just a tool to share code with others, it is a way to share code and associated metadata that is very useful if Something Goes Wrong (and it always does). Likewise docker hosts should be.

There is very real money in very real wallets on the line when hosts are created, so this is not something to be taken lightly.

I propose a docker hosts history command (or equivalent), as well as a mechanism for dry-runs with real-deal infrastructure (think Terraform's plan). Then you do your standard docker hosts create --driver ec2 kaiju, get back a graph of what will happen when you pull the trigger, and do the real thing with docker hosts apply (or whatever).

You could have a ~/.docker/hosts/foo/plan.json and/or ~/docker/hosts/foo/history.json to track these.

The way I envision this working is making drivers register their API requests (anything that mutates infra at least) and responses and considering it a bug if they don't. (all of my requests to the AWS API go through a common "gateway" function, so I know it would be pretty easy for me to implement but I don't know about other drivers) Alternatively we could try to implement some kind of base http.Client equivalent that logs stuff automatically.

Sync mechanism to keep hosts consistent across... hosts

This might entail the use of Docker Hub or a self-hosted mechanism, I'm not really sure. But most assuredly there needs to be a way to sync hosts across machines. The current implementation couples docker hosts extremely tightly to one machine (because it's all client side), if you use a different physical device, different VM, or server, you suddenly lose all of the stuff that went into creating your carefully curated flock of Docker hosts. And god forbid your hard drive catches on fire or something.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side

This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Figure out running commands against multiple hosts

I'm not sure exactly what form this will take (ability to select multiple active hosts?) - I'd prefer the community to throw in their 2c - but I do know that having access to only one host at a time will be a very big issue. Consider a basic infrastructure setup of three servers: two that run application code and a load balancer in front of the other two. To set this up (using Docker of course :)) with docker hosts in its current form I would have to switch the active host, pull the app image, switch the active host again, pull the app image, switch the active host again, and then pull the load balancer image. As much of this should be concurrent as possible or people just won't use it.

I'm super excited by this proposal and I encourage everyone to give the bfirsh/host-management branch a play (the Digital Ocean driver is probably pretty easy to get started with).

I have a few more things that I will add later as well 😄

Contributor

nathanleclaire commented Oct 22, 2014

@proppy Right now drivers implement this interface:

// Driver defines how a host is created and controlled. Different types of
// driver represent different ways hosts can be created (e.g. different
// hypervisors, different cloud providers)
type Driver interface {
    DriverName() string
    GetURL() (string, error)
    GetIP() (string, error)
    GetState() (state.State, error)
    Create() error
    SetConfigFromFlags(flags interface{}) error
    Remove() error
    Start() error
    Stop() error
    Restart() error
    Kill() error
    GetSSHCommand(args ...string) *exec.Cmd
    // Pause() error
}

and register themselves with a _ import in api/client/commands.go.

Feedback

First off, awesome work @bfirsh , @cpuguy83 and countless others who have been involved in this idea and moving this implementation forward. It's very exciting stuff.

I got interested in the movement around hosts and went off and implemented an Amazon EC2 driver. It's a lot of fun to be able to bootstrap a host pretty quickly from the command line.

Ben and the boot2docker-cli maintainers (which is where the original driver code came from) did a great job implementing an interface that's pretty easy to write to. Like anything new, there's plenty of rough surfaces that need to get ironed out before we even consider merging this into Docker.

My feedback on the proposal is as follows:

Implement docker hosts inspect foo

I've made a PR for this (bfirsh#4), I think it is a vital part of making an implementation of this succeed. Users shouldn't have to know of ~/.docker/hosts/myhost/config.json as an implementation detail.

Provide a way to track (and probably to plan) infrastructure changes

What is Docker about? Docker is about repeatability and about speed. If I can't take what I've created with Docker and repeat it elsewhere, fast, it is not much use to me. Therefore if Docker is going to get into the business of creating hosts to run Docker, it must provide a way to do so which is tracked and repeatable. Something like git is not just a tool to share code with others, it is a way to share code and associated metadata that is very useful if Something Goes Wrong (and it always does). Likewise docker hosts should be.

There is very real money in very real wallets on the line when hosts are created, so this is not something to be taken lightly.

I propose a docker hosts history command (or equivalent), as well as a mechanism for dry-runs with real-deal infrastructure (think Terraform's plan). Then you do your standard docker hosts create --driver ec2 kaiju, get back a graph of what will happen when you pull the trigger, and do the real thing with docker hosts apply (or whatever).

You could have a ~/.docker/hosts/foo/plan.json and/or ~/docker/hosts/foo/history.json to track these.

The way I envision this working is making drivers register their API requests (anything that mutates infra at least) and responses and considering it a bug if they don't. (all of my requests to the AWS API go through a common "gateway" function, so I know it would be pretty easy for me to implement but I don't know about other drivers) Alternatively we could try to implement some kind of base http.Client equivalent that logs stuff automatically.

Sync mechanism to keep hosts consistent across... hosts

This might entail the use of Docker Hub or a self-hosted mechanism, I'm not really sure. But most assuredly there needs to be a way to sync hosts across machines. The current implementation couples docker hosts extremely tightly to one machine (because it's all client side), if you use a different physical device, different VM, or server, you suddenly lose all of the stuff that went into creating your carefully curated flock of Docker hosts. And god forbid your hard drive catches on fire or something.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side

This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Figure out running commands against multiple hosts

I'm not sure exactly what form this will take (ability to select multiple active hosts?) - I'd prefer the community to throw in their 2c - but I do know that having access to only one host at a time will be a very big issue. Consider a basic infrastructure setup of three servers: two that run application code and a load balancer in front of the other two. To set this up (using Docker of course :)) with docker hosts in its current form I would have to switch the active host, pull the app image, switch the active host again, pull the app image, switch the active host again, and then pull the load balancer image. As much of this should be concurrent as possible or people just won't use it.

I'm super excited by this proposal and I encourage everyone to give the bfirsh/host-management branch a play (the Digital Ocean driver is probably pretty easy to get started with).

I have a few more things that I will add later as well 😄

@aanand

This comment has been minimized.

Show comment
Hide comment
@aanand

aanand Oct 22, 2014

Contributor

Implement docker hosts inspect foo

+1, good to see this merged already!

Provide a way to track (and probably to plan) infrastructure changes

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Sync mechanism to keep hosts consistent across... hosts
Figure out running commands against multiple hosts

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side
This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Not sure I understand the second use case here.

Contributor

aanand commented Oct 22, 2014

Implement docker hosts inspect foo

+1, good to see this merged already!

Provide a way to track (and probably to plan) infrastructure changes

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Sync mechanism to keep hosts consistent across... hosts
Figure out running commands against multiple hosts

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Move most of the code to a server-side API and implement a --bootstrap flag to optionally run it client-side
This takes care of the "install boot2docker using just the docker binary" use-case as well as the "I want to run docker hosts commands against a host that I created with docker hosts" fun stuff.

Not sure I understand the second use case here.

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Oct 24, 2014

Contributor

The rest of my feedback:

Don't switch active host by default on docker hosts create

Think git checkout vs. git checkout -b. I would have an optional flag to switch the host upon creation, since if anything went wrong with the creation of the host etc. it's well possible that the user's docker is now completely hosed and they probably won't be able to fix it themselves (and they shouldn't have to). Several times developing a driver I would get into awkward situations where my code messed up but the default host would change, so not only would I have to blow away ~/.docker/hosts and then delete the created EC2 instance manually, but then I would have to do things like docker -H=unix:///var/run/docker.sock hosts active default or else I would get panics. Fine for someone with deep docker knowledge, but not so fine for the average end user.

Don't assume the path is happy

It's situations like this that we should be prepared for:

  • What if the user's connection drops in the middle of host creation?
  • What if the API responds with something we didn't expect, like respond with a 503 due to them hammering it with requests? (this actually did happen to me while trying to spin up like 20 hosts at once to test my driver, haha)
  • What if the service messes up and mis-reports that an instance was created when it wasn't? etc.

Even if it's not our fault, we are the gateway and we will get blamed.

docker hosts -q doesn't work

This is probably not a big deal, but a small bug: -q works with docker hosts list and not with plain old docker hosts (and I almost always default to docker hosts without list since I'm lazy). This breaks the good ol' docker hosts rm $(docker hosts -q) idiom that people are used to from containers.

Consider making --driver blah an argument instead of a flag

So that basically, instead of :

docker hosts create --driver ec2 giantrobot

You have:

docker hosts create ec2 giantrobot

(this might actually help with the next point as well, since you could now try and figure out a way to chuck the driver-specific flags underneath the arg they belong to in docker hosts create ec2 -h or whatever)

Traditionally, UNIX flags have been for optional things, and arguments have been for required things; drivers are pretty close to required in our case (although --url for arbitrary hosts throws a weird wrench into things), so let's support them first class. Speaking of driverless hosts, what the heck happens when you run hosts commands on a host that was added without a driver? If we're going to support that, we must account for it.

Discuss how to clean up driver option flags

Right now it is a huge jungle of various flags for the various providers, and if/when we add more it will only get worse:

Usage: docker hosts create [OPTIONS] NAME

Create hosts

  --aws-access-key=""                                                                                    AWS Access Key
  --aws-image-id="ami-27939962"                                                                          AMI to use for the selected region
  --aws-instance-name=""                                                                                 Name of created instance
  --aws-instance-type="m1.small"                                                                         Type of instance to create
  --aws-instance-username="ubuntu"                                                                       Username for SSH on the instance (depends on AMI)
  --aws-region="us-west-1"                                                                               AWS Region
  --aws-secret-key=""                                                                                    AWS Secret Key
  --aws-security-group="docker-hosts"                                                                    Security group to use for the created instance
  --azure-docker-cert-dir=".docker"                                                                      Azure docker cert directory
  --azure-docker-port="4243"                                                                             Azure docker port
  --azure-image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB"    Azure image name
  --azure-location="West US"                                                                             Azure location
  --azure-name=""                                                                                        Azure name
  --azure-password=""                                                                                    Azure user password
  --azure-publish-settings-file=""                                                                       Azure publish settings file
  --azure-size="Small"                                                                                   Azure size
  --azure-ssh="22"                                                                                       Azure ssh port
  --azure-subscription-cert=""                                                                           Azure subscription cert
  --azure-subscription-id=""                                                                             Azure subscription ID
  --azure-username="tcuser"                                                                              Azure username
  -d, --driver="none"                                                                                    Driver to create host with. Available drivers: azure, digitalocean, ec2, none, virtualbox
  --digitalocean-access-token=""                                                                         Digital Ocean access token
  --digitalocean-image="docker"                                                                          Digital Ocean image
  --digitalocean-region="nyc3"                                                                           Digital Ocean region
  --digitalocean-size="512mb"                                                                            Digital Ocean size
  --no-install=false                                                                                     Do not install Docker during provisioning, assume it already exists
  --no-provision=false                                                                                   Do not provision the instance automatically
  --url=""                                                                                               URL of host when no driver is selected
  --virtualbox-boot2docker-url=""                                                                        The URL of the boot2docker image. Defaults to the latest available version
  --virtualbox-disk-size=20000                                                                           Size of disk for host in MB
  --virtualbox-memory=1024                                                                               Size of memory for host in MB

This is tough to parse quickly and e.g. having giant amounts of whitespace for everyone else because one of the azure defaults is really long is annoying. This should be cleaned up in the final implementation.

run and use instead of create and active commands, respectively

This one is kind of a minor UI nitpick, and I don't feel particularly strongly about it compared to some of the other points, but I would prefer docker hosts run --driver digitalocean foobar (since create in context of a container references both creating and starting one, just like we're doing here) with separate subcommands for create and start (and we have start already) just like docker. Additionally docker hosts use foobar would be nice since it's just easier to type and it is a verb.

docker hosts should report if the daemon is reachable or not, instead of just the instance status

This is one of those subtle things that you don't really think about until you get bitten by it, but I firmly believe that hosts should also have some kind of IsDaemonAvailable method to implement as well (it shouldn't be too bad - it will be something like a net.DialTimeout check on :2376 in most cases), and the results of that should be shown in docker hosts list. Otherwise things will race and if as an end user you assume that "host running" == "daemon available" it could be easy to screw yourself by not checking for the right thing.

Small footnote as well: there seems to not really be any enforced discipline around signing of the commits that are going into this branch, and that's going to really hurt a lot if we do end up re-using them.

Contributor

nathanleclaire commented Oct 24, 2014

The rest of my feedback:

Don't switch active host by default on docker hosts create

Think git checkout vs. git checkout -b. I would have an optional flag to switch the host upon creation, since if anything went wrong with the creation of the host etc. it's well possible that the user's docker is now completely hosed and they probably won't be able to fix it themselves (and they shouldn't have to). Several times developing a driver I would get into awkward situations where my code messed up but the default host would change, so not only would I have to blow away ~/.docker/hosts and then delete the created EC2 instance manually, but then I would have to do things like docker -H=unix:///var/run/docker.sock hosts active default or else I would get panics. Fine for someone with deep docker knowledge, but not so fine for the average end user.

Don't assume the path is happy

It's situations like this that we should be prepared for:

  • What if the user's connection drops in the middle of host creation?
  • What if the API responds with something we didn't expect, like respond with a 503 due to them hammering it with requests? (this actually did happen to me while trying to spin up like 20 hosts at once to test my driver, haha)
  • What if the service messes up and mis-reports that an instance was created when it wasn't? etc.

Even if it's not our fault, we are the gateway and we will get blamed.

docker hosts -q doesn't work

This is probably not a big deal, but a small bug: -q works with docker hosts list and not with plain old docker hosts (and I almost always default to docker hosts without list since I'm lazy). This breaks the good ol' docker hosts rm $(docker hosts -q) idiom that people are used to from containers.

Consider making --driver blah an argument instead of a flag

So that basically, instead of :

docker hosts create --driver ec2 giantrobot

You have:

docker hosts create ec2 giantrobot

(this might actually help with the next point as well, since you could now try and figure out a way to chuck the driver-specific flags underneath the arg they belong to in docker hosts create ec2 -h or whatever)

Traditionally, UNIX flags have been for optional things, and arguments have been for required things; drivers are pretty close to required in our case (although --url for arbitrary hosts throws a weird wrench into things), so let's support them first class. Speaking of driverless hosts, what the heck happens when you run hosts commands on a host that was added without a driver? If we're going to support that, we must account for it.

Discuss how to clean up driver option flags

Right now it is a huge jungle of various flags for the various providers, and if/when we add more it will only get worse:

Usage: docker hosts create [OPTIONS] NAME

Create hosts

  --aws-access-key=""                                                                                    AWS Access Key
  --aws-image-id="ami-27939962"                                                                          AMI to use for the selected region
  --aws-instance-name=""                                                                                 Name of created instance
  --aws-instance-type="m1.small"                                                                         Type of instance to create
  --aws-instance-username="ubuntu"                                                                       Username for SSH on the instance (depends on AMI)
  --aws-region="us-west-1"                                                                               AWS Region
  --aws-secret-key=""                                                                                    AWS Secret Key
  --aws-security-group="docker-hosts"                                                                    Security group to use for the created instance
  --azure-docker-cert-dir=".docker"                                                                      Azure docker cert directory
  --azure-docker-port="4243"                                                                             Azure docker port
  --azure-image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB"    Azure image name
  --azure-location="West US"                                                                             Azure location
  --azure-name=""                                                                                        Azure name
  --azure-password=""                                                                                    Azure user password
  --azure-publish-settings-file=""                                                                       Azure publish settings file
  --azure-size="Small"                                                                                   Azure size
  --azure-ssh="22"                                                                                       Azure ssh port
  --azure-subscription-cert=""                                                                           Azure subscription cert
  --azure-subscription-id=""                                                                             Azure subscription ID
  --azure-username="tcuser"                                                                              Azure username
  -d, --driver="none"                                                                                    Driver to create host with. Available drivers: azure, digitalocean, ec2, none, virtualbox
  --digitalocean-access-token=""                                                                         Digital Ocean access token
  --digitalocean-image="docker"                                                                          Digital Ocean image
  --digitalocean-region="nyc3"                                                                           Digital Ocean region
  --digitalocean-size="512mb"                                                                            Digital Ocean size
  --no-install=false                                                                                     Do not install Docker during provisioning, assume it already exists
  --no-provision=false                                                                                   Do not provision the instance automatically
  --url=""                                                                                               URL of host when no driver is selected
  --virtualbox-boot2docker-url=""                                                                        The URL of the boot2docker image. Defaults to the latest available version
  --virtualbox-disk-size=20000                                                                           Size of disk for host in MB
  --virtualbox-memory=1024                                                                               Size of memory for host in MB

This is tough to parse quickly and e.g. having giant amounts of whitespace for everyone else because one of the azure defaults is really long is annoying. This should be cleaned up in the final implementation.

run and use instead of create and active commands, respectively

This one is kind of a minor UI nitpick, and I don't feel particularly strongly about it compared to some of the other points, but I would prefer docker hosts run --driver digitalocean foobar (since create in context of a container references both creating and starting one, just like we're doing here) with separate subcommands for create and start (and we have start already) just like docker. Additionally docker hosts use foobar would be nice since it's just easier to type and it is a verb.

docker hosts should report if the daemon is reachable or not, instead of just the instance status

This is one of those subtle things that you don't really think about until you get bitten by it, but I firmly believe that hosts should also have some kind of IsDaemonAvailable method to implement as well (it shouldn't be too bad - it will be something like a net.DialTimeout check on :2376 in most cases), and the results of that should be shown in docker hosts list. Otherwise things will race and if as an end user you assume that "host running" == "daemon available" it could be easy to screw yourself by not checking for the right thing.

Small footnote as well: there seems to not really be any enforced discipline around signing of the commits that are going into this branch, and that's going to really hurt a lot if we do end up re-using them.

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Oct 24, 2014

Contributor

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Yes, I think this is absolute essential to implement and get correct now, or else people will not use this feature - we should log each mutating request and the received response, as well as have a staging / dry-run feature.

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

Fair enough, but what about a set of clients that share hosts? There needs to be some way to account for this. To continue the git comparison: you can easily set up remotes and track branches across different clients / machines, and I think users will want to run commands against the same hosts from different docker clients (I know I will).

Additionally, let's suppose that I have some SSH keys which were generated by docker hosts and are the only way to get into the created hosts on, say, my laptop: then I accidentally drop it into a puddle of water (or my hard drive gets corrupted somehow). With this implementation if that happens I am completely screwed.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Yes, let's not get into the business of aggregate host / resource scheduling etc. with this feature. But what's the answer to e.g. "how do I pull the same image concurrently on two or more hosts"? docker -H=myHost pull image & docker -H=myHost2 pull image?

Not sure I understand the second use case here.

I suppose what I was envisioning is exposing host management as an API, so that you could remotely run docker hosts commands. That way you could :

  • Support concurrent operations (mostly you can't run hosts create in the background right now because it's all client-side)
  • Run docker hosts commands on remote hosts themselves, creating a virtuous cycle and opening up the doors to take advantage of something like private IP addresses for additional speed etc.
  • Support docker hosts operations elegantly on 3rd-party clients like fig (valuable part of the ecosystem). No daemon API, no docker-py bindings (unless I'm mistaken and docker-py assumes the presence of a docker binary on the local system- and I very well could be).
Contributor

nathanleclaire commented Oct 24, 2014

This would be rad as hell. I'd argue that this isn't necessarily a day 1 feature - but on the other hand, it'll be harder to add later (drivers might have to be significantly rewritten).

Yes, I think this is absolute essential to implement and get correct now, or else people will not use this feature - we should log each mutating request and the received response, as well as have a staging / dry-run feature.

I think as soon as we're talking about a set of hosts that share an application, we're out of the scope of host management.

Fair enough, but what about a set of clients that share hosts? There needs to be some way to account for this. To continue the git comparison: you can easily set up remotes and track branches across different clients / machines, and I think users will want to run commands against the same hosts from different docker clients (I know I will).

Additionally, let's suppose that I have some SSH keys which were generated by docker hosts and are the only way to get into the created hosts on, say, my laptop: then I accidentally drop it into a puddle of water (or my hard drive gets corrupted somehow). With this implementation if that happens I am completely screwed.

If whatever system is making the decisions about what host to run what on, and what order to do things in, exposes the Docker API, then that could be a selectable host in docker hosts; otherwise I'm inclined to say it's none of our business.

Yes, let's not get into the business of aggregate host / resource scheduling etc. with this feature. But what's the answer to e.g. "how do I pull the same image concurrently on two or more hosts"? docker -H=myHost pull image & docker -H=myHost2 pull image?

Not sure I understand the second use case here.

I suppose what I was envisioning is exposing host management as an API, so that you could remotely run docker hosts commands. That way you could :

  • Support concurrent operations (mostly you can't run hosts create in the background right now because it's all client-side)
  • Run docker hosts commands on remote hosts themselves, creating a virtuous cycle and opening up the doors to take advantage of something like private IP addresses for additional speed etc.
  • Support docker hosts operations elegantly on 3rd-party clients like fig (valuable part of the ecosystem). No daemon API, no docker-py bindings (unless I'm mistaken and docker-py assumes the presence of a docker binary on the local system- and I very well could be).
@jeffmendoza

This comment has been minimized.

Show comment
Hide comment
@jeffmendoza

jeffmendoza Oct 27, 2014

Discuss how to clean up driver option flags

How about docker hosts create --help takes the --driver option? Not sure if this is easily doable with the current help infrastructure.

I like the idea of the driver being an option to the hosts create command from a user mindset pov. Having each driver create be a separate "command" encourages discontinuity.

For now, we'll remove the default option for the azure image, and behave like the b2d iso:

--virtualbox-boot2docker-url=""     The URL of the boot2docker image. Defaults to the latest available version

Discuss how to clean up driver option flags

How about docker hosts create --help takes the --driver option? Not sure if this is easily doable with the current help infrastructure.

I like the idea of the driver being an option to the hosts create command from a user mindset pov. Having each driver create be a separate "command" encourages discontinuity.

For now, we'll remove the default option for the azure image, and behave like the b2d iso:

--virtualbox-boot2docker-url=""     The URL of the boot2docker image. Defaults to the latest available version

@bfirsh bfirsh referenced this issue in boot2docker/boot2docker-cli Oct 28, 2014

Merged

Add Fusion driver and vmx template #265

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Oct 28, 2014

Contributor

@proppy Here are some examples of drivers: https://github.com/bfirsh/docker/tree/host-management/hosts/drivers

We plan to make this the primary installation experience for new users on OS X and we'll be deprecating boot2docker-cli. For Windows, there is ongoing work to add Windows support to the Docker client, so it depends if that lands or not.

Contributor

bfirsh commented Oct 28, 2014

@proppy Here are some examples of drivers: https://github.com/bfirsh/docker/tree/host-management/hosts/drivers

We plan to make this the primary installation experience for new users on OS X and we'll be deprecating boot2docker-cli. For Windows, there is ongoing work to add Windows support to the Docker client, so it depends if that lands or not.

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Oct 28, 2014

Contributor

@nathanleclaire Thanks for your pull requests Nathan, really appreciate it.

I agree we need a sync mechanism. Version 2 of host management will be a hosted service, probably as part of the Hub, but this is a first step to get us towards that. The migration path makes sense, I think. In the future version of Docker it'll ask you, "do you want to push your hosts up to the Hub?" (think iCloud migration).

With regards to selecting the host on create: I think this makes sense from a behaviour point of view. I think your objection is that sometimes it throws panics, and it shouldn't do that. It's a known issue that Docker won't when a stopped host is active, and I'm currently refactoring the client to make that work.

We should be using create, because that maps with docker create. I'm not 100% sold on "active", but I haven't come up with anything better yet. I like how docker hosts active prints the current active host, which is why I don't think use will work, but maybe that's not important. Easy to change, so I'll let this discussion continue!

Contributor

bfirsh commented Oct 28, 2014

@nathanleclaire Thanks for your pull requests Nathan, really appreciate it.

I agree we need a sync mechanism. Version 2 of host management will be a hosted service, probably as part of the Hub, but this is a first step to get us towards that. The migration path makes sense, I think. In the future version of Docker it'll ask you, "do you want to push your hosts up to the Hub?" (think iCloud migration).

With regards to selecting the host on create: I think this makes sense from a behaviour point of view. I think your objection is that sometimes it throws panics, and it shouldn't do that. It's a known issue that Docker won't when a stopped host is active, and I'm currently refactoring the client to make that work.

We should be using create, because that maps with docker create. I'm not 100% sold on "active", but I haven't come up with anything better yet. I like how docker hosts active prints the current active host, which is why I don't think use will work, but maybe that's not important. Easy to change, so I'll let this discussion continue!

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Oct 28, 2014

Contributor

In terms of cleaning up CLI commands, I went back and forth on this design with @tianon, so he might have opinions.

I don't think there's anything wrong with long help commands. In fact, I like them, because it makes it easy to get a quick overview of how the whole thing works. I made the Docker help much longer for this reason.

I don't think it makes sense for there to be separate commands for each driver. It's optional, for a start.

I think a good solution could be to group the options based on what driver they're for. It'll make the help text easier to scan and won't complicate the design.

/cc @cpuguy83

Contributor

bfirsh commented Oct 28, 2014

In terms of cleaning up CLI commands, I went back and forth on this design with @tianon, so he might have opinions.

I don't think there's anything wrong with long help commands. In fact, I like them, because it makes it easy to get a quick overview of how the whole thing works. I made the Docker help much longer for this reason.

I don't think it makes sense for there to be separate commands for each driver. It's optional, for a start.

I think a good solution could be to group the options based on what driver they're for. It'll make the help text easier to scan and won't complicate the design.

/cc @cpuguy83

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Nov 5, 2014

Contributor

I have added some builds to the description if you'd like to try it out.

Contributor

bfirsh commented Nov 5, 2014

I have added some builds to the description if you'd like to try it out.

@kojiromike

This comment has been minimized.

Show comment
Hide comment
@kojiromike

kojiromike Nov 6, 2014

Contributor

Er, just a quick sanity check to make sure I'm not missing something obvious. In the example you have

$ docker -H dev echo hello I am dev
hello I am dev

Is that supposed to be an image named echo on the host dev with whatever entrypoint (presumably echo) and the args I am dev? Because I don't see where you pulled or built an image named echo – I think you must've meant to run it on busybox.

Contributor

kojiromike commented Nov 6, 2014

Er, just a quick sanity check to make sure I'm not missing something obvious. In the example you have

$ docker -H dev echo hello I am dev
hello I am dev

Is that supposed to be an image named echo on the host dev with whatever entrypoint (presumably echo) and the args I am dev? Because I don't see where you pulled or built an image named echo – I think you must've meant to run it on busybox.

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Nov 6, 2014

Contributor

@kojiromike Oops. Yes. Thank you!

Contributor

bfirsh commented Nov 6, 2014

@kojiromike Oops. Yes. Thank you!

@inthecloud247

This comment has been minimized.

Show comment
Hide comment
@inthecloud247

inthecloud247 Nov 7, 2014

+1 Adding this as a separate tool would be great.

It feels irresponsible to add remote host functionality to core before the necessary security, permissions and logging are supported and stable.

Discussion of this functionality and similar comments by @nathanleclaire at 50+ minutes into: http://new.livestream.com/primeimagemedia/events/3532966

+1 Adding this as a separate tool would be great.

It feels irresponsible to add remote host functionality to core before the necessary security, permissions and logging are supported and stable.

Discussion of this functionality and similar comments by @nathanleclaire at 50+ minutes into: http://new.livestream.com/primeimagemedia/events/3532966

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Nov 13, 2014

Contributor

I've updated the binaries in the description with a more recent build.

Contributor

bfirsh commented Nov 13, 2014

I've updated the binaries in the description with a more recent build.

@adrianotto

This comment has been minimized.

Show comment
Hide comment
@adrianotto

adrianotto Nov 14, 2014

@inthecloud247 while we agree this is terrific functionality that Docker users will love to have, I disagree sharply with the idea that we should provide it as a separate tool. This feature set is so incredibly compelling that every docker user should have access to it built in. Yes, the default provider list will need to be carefully curated and managed, and this will cause docker to be larger. These drawbacks are well worth the advantage of making a simple and effective union between a local docker environment, a group of local docker servers, and even a collection of clouds where workloads can be run. Let's find a way to get at least support providers who can assign maintainers to provide for the long term care and feeding of their contributions.

@inthecloud247 while we agree this is terrific functionality that Docker users will love to have, I disagree sharply with the idea that we should provide it as a separate tool. This feature set is so incredibly compelling that every docker user should have access to it built in. Yes, the default provider list will need to be carefully curated and managed, and this will cause docker to be larger. These drawbacks are well worth the advantage of making a simple and effective union between a local docker environment, a group of local docker servers, and even a collection of clouds where workloads can be run. Let's find a way to get at least support providers who can assign maintainers to provide for the long term care and feeding of their contributions.

@inthecloud247

This comment has been minimized.

Show comment
Hide comment
@inthecloud247

inthecloud247 Nov 14, 2014

Some of the multi-host functionality seems to be available now using fig... just need to merge the changes proposed by the recent Docker Global Hack Day 2 winner. The pull request has been open for 11 days:

docker/compose#607
https://blog.docker.com/2014/11/announcing-docker-global-hack-day-2-winners/

Kinda sad if he wins the hack day and the pull request gets closed :-(

Some of the multi-host functionality seems to be available now using fig... just need to merge the changes proposed by the recent Docker Global Hack Day 2 winner. The pull request has been open for 11 days:

docker/compose#607
https://blog.docker.com/2014/11/announcing-docker-global-hack-day-2-winners/

Kinda sad if he wins the hack day and the pull request gets closed :-(

@proppy

This comment has been minimized.

Show comment
Hide comment
@proppy

proppy Nov 21, 2014

Contributor

Does the driver expect to be able control the daemon? or is that cool if the daemon start with the VM?

Also if the VM generates the certs at boot, it there a standard way for the hosts code to pull the client certs over ssh? or should it be done per driver?

Contributor

proppy commented Nov 21, 2014

Does the driver expect to be able control the daemon? or is that cool if the daemon start with the VM?

Also if the VM generates the certs at boot, it there a standard way for the hosts code to pull the client certs over ssh? or should it be done per driver?

@jeffmendoza

This comment has been minimized.

Show comment
Hide comment
@jeffmendoza

jeffmendoza Nov 21, 2014

@proppy The driver just needs to boot up the vm with the daemon listening on an available port. Weather that means installing the daemon or having docker baked into the image, it doesn't matter.

None of the drivers have security at the moment, we're moving away from the CA based certs to ssh style: #8265

I'd like to get an interface into that PR for the driver to be able to pull the clients 'public key,' so that the driver can properly configure the deamons 'authorized_clients' file, but haven't had a chance. We'll need this when these features merge.

@proppy The driver just needs to boot up the vm with the daemon listening on an available port. Weather that means installing the daemon or having docker baked into the image, it doesn't matter.

None of the drivers have security at the moment, we're moving away from the CA based certs to ssh style: #8265

I'd like to get an interface into that PR for the driver to be able to pull the clients 'public key,' so that the driver can properly configure the deamons 'authorized_clients' file, but haven't had a chance. We'll need this when these features merge.

@frapposelli

This comment has been minimized.

Show comment
Hide comment
@frapposelli

frapposelli Nov 24, 2014

Contributor

Love this proposal 👍

I just filed an issue over at bfirsh/docker#18 as we have several VMware drivers ready to be PR'd (vCloud Air, Fusion and vSphere), can you guys shed some light on what are the plans for this proposal?

Contributor

frapposelli commented Nov 24, 2014

Love this proposal 👍

I just filed an issue over at bfirsh/docker#18 as we have several VMware drivers ready to be PR'd (vCloud Air, Fusion and vSphere), can you guys shed some light on what are the plans for this proposal?

@proppy

This comment has been minimized.

Show comment
Hide comment
@proppy

proppy Nov 24, 2014

Contributor

@jeffmendoza does the driver assume it has lifecycle control over the daemon?

In the current spec does:

    Start() error
    Stop() error
    Restart() error

refers to the host or the daemon? or it doesn't matter?

Contributor

proppy commented Nov 24, 2014

@jeffmendoza does the driver assume it has lifecycle control over the daemon?

In the current spec does:

    Start() error
    Stop() error
    Restart() error

refers to the host or the daemon? or it doesn't matter?

@jeffmendoza

This comment has been minimized.

Show comment
Hide comment
@jeffmendoza

jeffmendoza Nov 24, 2014

@proppy Those refer to the host. Example: you would want to use docker hosts stop to save money on your public cloud based host.

@proppy Those refer to the host. Example: you would want to use docker hosts stop to save money on your public cloud based host.

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Nov 24, 2014

Contributor

For those working on host management, just a heads up: I have started to squash my branch and base it on top of #8265. The new branch is here: https://github.com/bfirsh/docker/tree/host-management-with-client-daemon-auth

I will transfer any changes from the host-management branch over to that branch, but if you start building on top of that branch now, the merge conflicts might be less painful. :)

Contributor

bfirsh commented Nov 24, 2014

For those working on host management, just a heads up: I have started to squash my branch and base it on top of #8265. The new branch is here: https://github.com/bfirsh/docker/tree/host-management-with-client-daemon-auth

I will transfer any changes from the host-management branch over to that branch, but if you start building on top of that branch now, the merge conflicts might be less painful. :)

@razic

This comment has been minimized.

Show comment
Hide comment
@razic

razic Dec 1, 2014

Man docker is moving so fast.

razic commented Dec 1, 2014

Man docker is moving so fast.

@dlorenc dlorenc referenced this issue in bfirsh/docker Dec 2, 2014

Closed

GCE hosts driver. #21

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Dec 2, 2014

Contributor

I have now rebased the host management branch on top of #8265 and squashed it:

https://github.com/bfirsh/docker/compare/host-management

Any pull requests should now be based on top of that. The driver interface hasn't changed, so it should be a trivial matter to rebase any existing pull requests. The main thing which has changed is that drivers are expected to set up identity auth for communication with the host. See this commit for an example of how to do so.

The old branch is here for reference.

Full update and preview builds coming soon.

Contributor

bfirsh commented Dec 2, 2014

I have now rebased the host management branch on top of #8265 and squashed it:

https://github.com/bfirsh/docker/compare/host-management

Any pull requests should now be based on top of that. The driver interface hasn't changed, so it should be a trivial matter to rebase any existing pull requests. The main thing which has changed is that drivers are expected to set up identity auth for communication with the host. See this commit for an example of how to do so.

The old branch is here for reference.

Full update and preview builds coming soon.

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Dec 3, 2014

Member

Has there been any progress on splitting the actual driver implementations
out of the core binary?

Member

tianon commented Dec 3, 2014

Has there been any progress on splitting the actual driver implementations
out of the core binary?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 3, 2014

Contributor

@tianon Not really official sanctioned, but I've been playing around with implementing plugins, specifically implenenting a graphdriver for plugins, which could be used for this, in theory: https://github.com/cpuguy83/docker/tree/poc_storage_plugin

Contributor

cpuguy83 commented Dec 3, 2014

@tianon Not really official sanctioned, but I've been playing around with implementing plugins, specifically implenenting a graphdriver for plugins, which could be used for this, in theory: https://github.com/cpuguy83/docker/tree/poc_storage_plugin

@bfirsh

This comment has been minimized.

Show comment
Hide comment
@bfirsh

bfirsh Dec 4, 2014

Contributor

Thanks for your feedback, everyone. Host management is now Docker Machine!

https://github.com/docker/machine

Contributor

bfirsh commented Dec 4, 2014

Thanks for your feedback, everyone. Host management is now Docker Machine!

https://github.com/docker/machine

@bfirsh bfirsh closed this Dec 4, 2014

@inthecloud247

This comment has been minimized.

Show comment
Hide comment
@inthecloud247

inthecloud247 Dec 4, 2014

Silly me, I assumed we were having an open discussion here... Secrets within puzzles inside a mystery.

It's been real.

Silly me, I assumed we were having an open discussion here... Secrets within puzzles inside a mystery.

It's been real.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment