Add Swarm management backend #23362

Merged
merged 76 commits into from Jun 14, 2016

Projects

None yet
@tonistiigi
Contributor
tonistiigi commented Jun 8, 2016 edited

As described in our ROADMAP.md, introduce new Swarm management API endpoints relying on docker/swarmkit to deploy services. It currently vendors docker/engine-api changes.

This PR is fully backward compatible (joining a Swarm is an optional feature of the Engine, and existing commands are not impacted).

Signed-off-by: Tonis Tiigi tonistiigi@gmail.com
Signed-off-by: Victor Vieux vieux@docker.com
Signed-off-by: Daniel Nephin dnephin@docker.com

@icecrime icecrime added this to the 1.12.0 milestone Jun 8, 2016
@thaJeztah
Member

Looks like some tests are failing

@aluzzardi aluzzardi commented on an outdated diff Jun 12, 2016
docs/reference/api/docker_remote_api_v1.24.md
+- **500** – server error
+
+### Update a service
+
+(TODO)
+
+## 3.9 Tasks
+
+**Note:** Tasks operations require to first be part of a Swarm.
+
+### List tasks
+
+**Warning:** this endpoint is part of the Swarm management feature introduced in Docker 1.12, and
+might be subject to non backward-compatible changes.
+
+`GET /swarm/tasks`
@aluzzardi
aluzzardi Jun 12, 2016 Contributor

Should be GET /tasks

/cc @icecrime

@mountkin mountkin and 1 other commented on an outdated diff Jun 12, 2016
daemon/cluster/convert/swarm.go
+ swarm.Spec.CAConfig.NodeCertExpiry, _ = ptypes.Duration(c.Spec.CAConfig.NodeCertExpiry)
+
+ // Meta
+ swarm.Version.Index = c.Meta.Version.Index
+ swarm.CreatedAt, _ = ptypes.Timestamp(c.Meta.CreatedAt)
+ swarm.UpdatedAt, _ = ptypes.Timestamp(c.Meta.UpdatedAt)
+
+ // Annotations
+ swarm.Spec.Name = c.Spec.Annotations.Name
+ swarm.Spec.Labels = c.Spec.Annotations.Labels
+
+ for _, policy := range c.Spec.AcceptancePolicy.Policies {
+ swarm.Spec.AcceptancePolicy.Policies = append(swarm.Spec.AcceptancePolicy.Policies, types.Policy{
+ Role: types.NodeRole(policy.Role.String()),
+ Autoaccept: policy.Autoaccept,
+ Secret: string(policy.Secret.Data),
@mountkin
mountkin Jun 12, 2016 Contributor

Panicked here when running docker swarm inspect.

2016-06-12 15:02:57.184744 I | http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 234 [running]:
net/http.(*conn).serve.func1(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x1a6c400, 0xc820010070)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/docker/docker/daemon/cluster/convert.SwarmFromGRPC(0xc820ee0280, 0x19, 0x9, 0xc8208c5e40, 0xc8208c5e50, 0xc8208c5e11, 0x7, 0x0, 0xc8208c5ea0, 0x2, ...)
        /go/src/github.com/docker/docker/daemon/cluster/convert/swarm.go:50 +0x267
github.com/docker/docker/daemon/cluster.(*Cluster).Inspect(0xc8200b9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/docker/docker/daemon/cluster/cluster.go:411 +0x23f
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).inspectCluster(0xc82072a7e0, 0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster_routes.go:48 +0x67
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).(github.com/docker/docker/api/server/router/swarm.inspectCluster)-fm(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster.go:30 +0x74
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x9aa
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089d980, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x68e
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x7f350b7c5940, 0xc8200110f8, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/debug.go:21 +0xd4d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/server.go:139 +0x138
net/http.HandlerFunc.ServeHTTP(0xc820859820, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc82000d0e0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc8206c71a0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
net/http.serverHandler.ServeHTTP(0xc8202a7f80, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2137 +0x44e

Steps to reproduce:

  1. Start docker service with /usr/bin/dockerd --debug --raw-logs
  2. docker swarm init
  3. docker swarm inspect
@vdemeester
vdemeester Jun 12, 2016 Member

Same here 😓

root@020824023c36:/go/src/github.com/docker/docker# docker swarm inspect
[]
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/swarm: EOF
tonistiigi and others added some commits Jun 7, 2016
@tonistiigi tonistiigi Support force new cluster
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
7e67984
@tonistiigi tonistiigi Add Swarm management backend
As described in our ROADMAP.md, introduce new Swarm management API
endpoints relying on swarmkit to deploy services. It currently vendors
docker/engine-api changes.

This PR is fully backward compatible (joining a Swarm is an optional
feature of the Engine, and existing commands are not impacted).

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
Signed-off-by: Daniel Nephin <dnephin@docker.com>
129b3a6
@tonistiigi tonistiigi Swarm integration tests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
fa92a2d
@icecrime @tonistiigi icecrime Add swarm api reference docs
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
f7e554c
@tonistiigi tonistiigi Set acceptance policy synchronously in init
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
f42b2dd
@tonistiigi tonistiigi Remove daemon conf dir override
Not needed because no tests using the key
use multiple daemons. Can be added separately.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
958a605
@tonistiigi tonistiigi Test updates
- Set reconciliation timeout as a variable
- Add timeout to shutdown and increase test timeout to include time needed
for cluster component shutdown.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
14d9fde
@cpuguy83 @tonistiigi cpuguy83 Fix panic if daemon is stopped before api is ready
This is easily visibile when the daemon is part of a swarm, and starting
the daemon before raft has finish initializing (which can take some
time).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
b6b3125
@vieux @tonistiigi vieux add CACertHash: backend
Signed-off-by: Victor Vieux <vieux@docker.com>
eb82d5b
@vieux @tonistiigi vieux fix panic @vieux introduced
Signed-off-by: Victor Vieux <vieux@docker.com>
e349508
@jpetazzo @tonistiigi jpetazzo Minor typo fix
Signed-off-by: Jérôme Petazzoni <jerome.petazzoni@gmail.com>
6eb671d
@tonistiigi tonistiigi Update tests
- Manual acceptance
- Promote/demote
- Raft timeouts

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
6e508af
@tonistiigi tonistiigi Update integration to latest swarmkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2a3f03a
@dnephin @tonistiigi dnephin Fix CLI side of volume problem. (backend)
Signed-off-by: Daniel Nephin <dnephin@docker.com>
b607c90
@vieux @tonistiigi vieux update to newest engine-api (backend)
Signed-off-by: Victor Vieux <vieux@docker.com>
3b9e6f0
@icecrime @tonistiigi icecrime Rename `SwarmPort` to `PublicPort`
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
5eaf2fc
@tonistiigi tonistiigi Remove rename test fix
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
7bf1e4d
@tonistiigi tonistiigi Add test for CAHash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
c2404e9
@mavenugo @tonistiigi mavenugo Make swarm-mode incompatible with --cluster-store configurations
In order to take care of init/leave/init cases, we need a reliable
way to reset the ClusterProvider and hence exposed the
SetClusterProvider Daemon API via the backend interface.
With this change, we handle the various combinations of swarm init and
--cluster-store configurations to be reliably detect as incompatible.

Signed-off-by: Madhu Venugopal <madhu@docker.com>
3a229e7
@vieux @tonistiigi vieux removed fields from engine-api (backend)
Signed-off-by: Victor Vieux <vieux@docker.com>
2cf43b8
@tonistiigi tonistiigi Fix rolling update on upscaling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
e746280
@mavenugo @tonistiigi mavenugo Avoid persisting dynamically created network
Signed-off-by: Madhu Venugopal <madhu@docker.com>
ca1c47d
@tonistiigi tonistiigi Fix possible deadlocks
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
82ddf5d
@tonistiigi tonistiigi Skipped stopped container on shutdown
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
42690f9
@vieux @tonistiigi vieux upadate engine-api: backend
Signed-off-by: Victor Vieux <vieux@docker.com>
54d6126
@tonistiigi tonistiigi Update integration to new swarmkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
4d67547
@vieux @tonistiigi vieux fix compilation
Signed-off-by: Victor Vieux <vieux@docker.com>
c5198b1
@vieux @tonistiigi vieux fix panic in secret
Signed-off-by: Victor Vieux <vieux@docker.com>
7030903
@mavenugo @tonistiigi mavenugo Prevent unmanaged containers to use swarm managed networks
Signed-off-by: Madhu Venugopal <madhu@docker.com>
b4a53a1
@mavenugo @tonistiigi mavenugo accept partial-id for `docker network rm`
Signed-off-by: Madhu Venugopal <madhu@docker.com>
ef19cb3
@tonistiigi tonistiigi Update integration to new swarmkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
c7f604e
@tonistiigi tonistiigi Fix flakiness on container shutdown
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
32a4a57
@mrjana @tonistiigi mrjana Recover from stale ingress network plumbing
Currently on cluster ungraceful shutdown ingrss network plumbing can
be left behind and on restart we need to properly cleanup the stale
plumbing to make it useful.

Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
93c6503
@vieux @tonistiigi vieux fix case
Signed-off-by: Victor Vieux <vieux@docker.com>
dbc78aa
@vieux @tonistiigi vieux backend
Signed-off-by: Victor Vieux <vieux@docker.com>
d5cb438
@vieux @tonistiigi vieux fix mounts
Signed-off-by: Victor Vieux <vieux@docker.com>
d17b30b
@vieux @tonistiigi vieux backend
Signed-off-by: Victor Vieux <vieux@docker.com>
95ff9f4
@tonistiigi tonistiigi Fix tasks endpoint in reference docs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
a1aef23
@mavenugo @tonistiigi mavenugo swarm managed networks should me marked as "swarm" scoped (backend)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
6e0bbc1
@mrjana @tonistiigi mrjana Vendoring libnetwork @892e1b9 (backend)
Adds overlay datapath security enhancement
Adds netlink performance enhancements
Bug fixes

Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
d683992
@tonistiigi tonistiigi Remove state on failed init
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
a7d0b2c
@mrjana @tonistiigi mrjana Do not fail on predefined network for managed network
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
3af6ddc
@mrjana @tonistiigi mrjana Update backend to libnetwork @e8da32c
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
a1ddd60
@dnephin @tonistiigi dnephin Add Swarm management CLI commands
As described in our ROADMAP.md, introduce new Swarm management commands
to call to the corresponding API endpoints.

This PR is fully backward compatible (joining a Swarm is an optional
feature of the Engine, and existing commands are not impacted).

Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
bdcd802
@amitshukla @tonistiigi amitshukla CLI docs
Signed-off-by: Amit Shukla <amit.shukla@docker.com>
1cac395
@icecrime @tonistiigi icecrime Shorten `docker service` restart policy flags
Rename `--restart-policy-*` flags to `--restart-*` (e.g.,
`--restart-policy-delay` becomes `--restart-delay`).

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
ab06a5b
@icecrime @tonistiigi icecrime Clarify services update policies
Rename `--updateconfig-*` to `--update-*` and improve help messages.

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
0c84337
@vieux @tonistiigi vieux use ID for '*'
Signed-off-by: Victor Vieux <vieux@docker.com>
b301d2c
@vieux @tonistiigi vieux add CACertHash: cli
Signed-off-by: Victor Vieux <vieux@docker.com>
ddb578c
@tonistiigi tonistiigi Fix accept policy initialization
- Set secret when using default policy
- Always initialize policy for all roles

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
89a5556
@tonistiigi tonistiigi Update join command message
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
c5c1fc5
@dnephin @tonistiigi dnephin Fix CLI side of volume problem.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
713cbd8
@vieux @tonistiigi vieux update to newest engine-api (cli)
Signed-off-by: Victor Vieux <vieux@docker.com>
412aa1e
@icecrime @tonistiigi icecrime Rename `--port` to `--publish` in `docker service`
Rename `--port` to `--publish` for consistency between `docker service`
and `docker run`.

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
d5745c8
@tonistiigi tonistiigi Fix RenameCheckNames test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
6d35ce7
@dnephin @tonistiigi dnephin Fixes #273 - set command line args in args not command
Signed-off-by: Daniel Nephin <dnephin@docker.com>
f798f43
@dnephin @tonistiigi dnephin backport fixes from docker/docker PR comments.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
bc81ad5
@vieux @tonistiigi vieux upadate engine-api: cli
Signed-off-by: Victor Vieux <vieux@docker.com>
f6dd2a5
@icecrime @tonistiigi icecrime Pretty print strings on human-readable outputs
Capitalize first letter in output meant for human consumptions (e.g.,
`docker service ls`, or `docker node inspect --pretty`).

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
fb33899
@icecrime @tonistiigi icecrime Fix incorrect `docker service inspect` help message
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
5bd2b91
@vieux @tonistiigi vieux fix docker swarm inspect output on error
Signed-off-by: Victor Vieux <vieux@docker.com>
bee98b5
@vieux @tonistiigi vieux git rm reject.go
Signed-off-by: Victor Vieux <vieux@docker.com>
740a118
@vieux @tonistiigi vieux add -h back to inspect
Signed-off-by: Victor Vieux <vieux@docker.com>
2dac952
@vieux @tonistiigi vieux cli
Signed-off-by: Victor Vieux <vieux@docker.com>
3d7b63a
@vieux @tonistiigi vieux api
Signed-off-by: Victor Vieux <vieux@docker.com>
654bf01
@mavenugo @tonistiigi mavenugo swarm managed networks should me marked as "swarm" scoped (cli)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
484b556
@icecrime @tonistiigi icecrime Add `docker service scale` command
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
b3eefed
@mrjana @tonistiigi mrjana Vendoring libnetwork @892e1b9 (cli)
Adds overlay datapath security enhancement
Adds netlink performance enhancements
Bug fixes

Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
8ae22c0
@dongluochen @tonistiigi dongluochen Upate auto-accept help text.
Signed-off-by: Dong Chen <dongluo.chen@docker.com>
b4ba9c6
@dnephin @tonistiigi dnephin Small UX improvements to 'docker service scale'
Signed-off-by: Daniel Nephin <dnephin@docker.com>
5d07eb3
@icecrime @tonistiigi icecrime Add missing newline
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
9571a23
@cpuguy83 @tonistiigi cpuguy83 Fix scale pointer value in service inspect -p
Before this change, pretty output shows `859533555312` instead of `1`.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
5259aa0
@dperny @tonistiigi dperny Fix command not displaying in service ls.
Closes #396.

Signed-off-by: Drew Erny <drew.erny@docker.com>
dc0b20d
@dnephin @tonistiigi dnephin New format for setting mounts on a service.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
068354d
@dnephin @tonistiigi dnephin Upate from PR review.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
53c7216
@vieux @tonistiigi vieux fix rebase
Signed-off-by: Victor Vieux <vieux@docker.com>
6f5d29f
@icecrime
Member

LGTM 💯

@mavenugo
Contributor

🎉 🎉 LGTM

@vieux
Member
vieux commented Jun 14, 2016

LGTM 💯 this is very good!

@icecrime
Member

API reference will be completed in a followup 👍

@GordonTheTurtle GordonTheTurtle added dco/no and removed dco/no labels Jun 14, 2016
@icecrime icecrime merged commit 9646d17 into swarm Jun 14, 2016

1 of 8 checks passed

documentation started
Details
experimental Jenkins build Docker-PRs-experimental 19905 is running
Details
gccgo Jenkins build Docker-PRs-gccgo 6772 is running
Details
janky Jenkins build Docker-PRs 28699 is running
Details
userns Jenkins build Docker-PRs-userns 10861 is running
Details
win2lin Jenkins build Docker-PRs-Win2Lin 27285 is running
Details
windowsTP5 Jenkins build Docker-PRs-WoW-TP5 3190 is running
Details
docker/dco-signed All commits signed
Details
@vieux vieux deleted the swarm-backend branch Jul 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment