New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Swarm management backend #23362

Merged
merged 76 commits into from Jun 14, 2016

Conversation

Projects
None yet
@tonistiigi
Member

tonistiigi commented Jun 8, 2016

As described in our ROADMAP.md, introduce new Swarm management API endpoints relying on docker/swarmkit to deploy services. It currently vendors docker/engine-api changes.

This PR is fully backward compatible (joining a Swarm is an optional feature of the Engine, and existing commands are not impacted).

Signed-off-by: Tonis Tiigi tonistiigi@gmail.com
Signed-off-by: Victor Vieux vieux@docker.com
Signed-off-by: Daniel Nephin dnephin@docker.com

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jun 8, 2016

Member

Looks like some tests are failing

Member

thaJeztah commented Jun 8, 2016

Looks like some tests are failing

Show outdated Hide outdated docs/reference/api/docker_remote_api_v1.24.md
**Warning:** this endpoint is part of the Swarm management feature introduced in Docker 1.12, and
might be subject to non backward-compatible changes.
`GET /swarm/tasks`

This comment has been minimized.

@aluzzardi

aluzzardi Jun 12, 2016

Contributor

Should be GET /tasks

/cc @icecrime

@aluzzardi

aluzzardi Jun 12, 2016

Contributor

Should be GET /tasks

/cc @icecrime

Show outdated Hide outdated daemon/cluster/convert/swarm.go
swarm.Spec.AcceptancePolicy.Policies = append(swarm.Spec.AcceptancePolicy.Policies, types.Policy{
Role: types.NodeRole(policy.Role.String()),
Autoaccept: policy.Autoaccept,
Secret: string(policy.Secret.Data),

This comment has been minimized.

@mountkin

mountkin Jun 12, 2016

Contributor

Panicked here when running docker swarm inspect.

2016-06-12 15:02:57.184744 I | http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 234 [running]:
net/http.(*conn).serve.func1(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x1a6c400, 0xc820010070)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/docker/docker/daemon/cluster/convert.SwarmFromGRPC(0xc820ee0280, 0x19, 0x9, 0xc8208c5e40, 0xc8208c5e50, 0xc8208c5e11, 0x7, 0x0, 0xc8208c5ea0, 0x2, ...)
        /go/src/github.com/docker/docker/daemon/cluster/convert/swarm.go:50 +0x267
github.com/docker/docker/daemon/cluster.(*Cluster).Inspect(0xc8200b9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/docker/docker/daemon/cluster/cluster.go:411 +0x23f
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).inspectCluster(0xc82072a7e0, 0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster_routes.go:48 +0x67
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).(github.com/docker/docker/api/server/router/swarm.inspectCluster)-fm(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster.go:30 +0x74
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x9aa
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089d980, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x68e
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x7f350b7c5940, 0xc8200110f8, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/debug.go:21 +0xd4d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/server.go:139 +0x138
net/http.HandlerFunc.ServeHTTP(0xc820859820, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc82000d0e0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc8206c71a0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
net/http.serverHandler.ServeHTTP(0xc8202a7f80, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2137 +0x44e

Steps to reproduce:

  1. Start docker service with /usr/bin/dockerd --debug --raw-logs
  2. docker swarm init
  3. docker swarm inspect
@mountkin

mountkin Jun 12, 2016

Contributor

Panicked here when running docker swarm inspect.

2016-06-12 15:02:57.184744 I | http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 234 [running]:
net/http.(*conn).serve.func1(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x1a6c400, 0xc820010070)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/docker/docker/daemon/cluster/convert.SwarmFromGRPC(0xc820ee0280, 0x19, 0x9, 0xc8208c5e40, 0xc8208c5e50, 0xc8208c5e11, 0x7, 0x0, 0xc8208c5ea0, 0x2, ...)
        /go/src/github.com/docker/docker/daemon/cluster/convert/swarm.go:50 +0x267
github.com/docker/docker/daemon/cluster.(*Cluster).Inspect(0xc8200b9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/docker/docker/daemon/cluster/cluster.go:411 +0x23f
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).inspectCluster(0xc82072a7e0, 0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster_routes.go:48 +0x67
github.com/docker/docker/api/server/router/swarm.(*swarmRouter).(github.com/docker/docker/api/server/router/swarm.inspectCluster)-fm(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/router/swarm/cluster.go:30 +0x74
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089db30, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x9aa
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x7f350b7d37d8, 0xc82089d980, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x68e
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x7f350b7c5940, 0xc8200110f8, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0, 0xc82089d6e0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/server/middleware/debug.go:21 +0xd4d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/server.go:139 +0x138
net/http.HandlerFunc.ServeHTTP(0xc820859820, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc82000d0e0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc8206c71a0, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
net/http.serverHandler.ServeHTTP(0xc8202a7f80, 0x7f3508607e78, 0xc8203fa680, 0xc8214a5dc0)
        /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc8203c5b80)
        /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2137 +0x44e

Steps to reproduce:

  1. Start docker service with /usr/bin/dockerd --debug --raw-logs
  2. docker swarm init
  3. docker swarm inspect

This comment has been minimized.

@vdemeester

vdemeester Jun 12, 2016

Member

Same here 😓

root@020824023c36:/go/src/github.com/docker/docker# docker swarm inspect
[]
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/swarm: EOF
@vdemeester

vdemeester Jun 12, 2016

Member

Same here 😓

root@020824023c36:/go/src/github.com/docker/docker# docker swarm inspect
[]
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/swarm: EOF

tonistiigi and others added some commits Jun 7, 2016

Support force new cluster
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add Swarm management backend
As described in our ROADMAP.md, introduce new Swarm management API
endpoints relying on swarmkit to deploy services. It currently vendors
docker/engine-api changes.

This PR is fully backward compatible (joining a Swarm is an optional
feature of the Engine, and existing commands are not impacted).

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Swarm integration tests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Victor Vieux <vieux@docker.com>
Add swarm api reference docs
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Set acceptance policy synchronously in init
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Remove daemon conf dir override
Not needed because no tests using the key
use multiple daemons. Can be added separately.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Test updates
- Set reconciliation timeout as a variable
- Add timeout to shutdown and increase test timeout to include time needed
for cluster component shutdown.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fix panic if daemon is stopped before api is ready
This is easily visibile when the daemon is part of a swarm, and starting
the daemon before raft has finish initializing (which can take some
time).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
add CACertHash: backend
Signed-off-by: Victor Vieux <vieux@docker.com>
fix panic @vieux introduced
Signed-off-by: Victor Vieux <vieux@docker.com>
Minor typo fix
Signed-off-by: Jérôme Petazzoni <jerome.petazzoni@gmail.com>
Update tests
- Manual acceptance
- Promote/demote
- Raft timeouts

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Update integration to latest swarmkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fix CLI side of volume problem. (backend)
Signed-off-by: Daniel Nephin <dnephin@docker.com>
update to newest engine-api (backend)
Signed-off-by: Victor Vieux <vieux@docker.com>
Rename `SwarmPort` to `PublicPort`
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Remove rename test fix
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add test for CAHash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Make swarm-mode incompatible with --cluster-store configurations
In order to take care of init/leave/init cases, we need a reliable
way to reset the ClusterProvider and hence exposed the
SetClusterProvider Daemon API via the backend interface.
With this change, we handle the various combinations of swarm init and
--cluster-store configurations to be reliably detect as incompatible.

Signed-off-by: Madhu Venugopal <madhu@docker.com>
removed fields from engine-api (backend)
Signed-off-by: Victor Vieux <vieux@docker.com>
Fix rolling update on upscaling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Avoid persisting dynamically created network
Signed-off-by: Madhu Venugopal <madhu@docker.com>
Fix possible deadlocks
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Skipped stopped container on shutdown
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
upadate engine-api: backend
Signed-off-by: Victor Vieux <vieux@docker.com>
Update integration to new swarmkit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
fix compilation
Signed-off-by: Victor Vieux <vieux@docker.com>

Arnaud Porterie (icecrime) and others added some commits Jun 9, 2016

Rename `--port` to `--publish` in `docker service`
Rename `--port` to `--publish` for consistency between `docker service`
and `docker run`.

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Fix RenameCheckNames test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fixes #273 - set command line args in args not command
Signed-off-by: Daniel Nephin <dnephin@docker.com>
backport fixes from docker/docker PR comments.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
upadate engine-api: cli
Signed-off-by: Victor Vieux <vieux@docker.com>
Pretty print strings on human-readable outputs
Capitalize first letter in output meant for human consumptions (e.g.,
`docker service ls`, or `docker node inspect --pretty`).

Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Fix incorrect `docker service inspect` help message
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
fix docker swarm inspect output on error
Signed-off-by: Victor Vieux <vieux@docker.com>
git rm reject.go
Signed-off-by: Victor Vieux <vieux@docker.com>
add -h back to inspect
Signed-off-by: Victor Vieux <vieux@docker.com>
cli
Signed-off-by: Victor Vieux <vieux@docker.com>
api
Signed-off-by: Victor Vieux <vieux@docker.com>
swarm managed networks should me marked as "swarm" scoped (cli)
Signed-off-by: Madhu Venugopal <madhu@docker.com>
Add `docker service scale` command
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Vendoring libnetwork @892e1b9864d1f (cli)
Adds overlay datapath security enhancement
Adds netlink performance enhancements
Bug fixes

Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
Upate auto-accept help text.
Signed-off-by: Dong Chen <dongluo.chen@docker.com>
Small UX improvements to 'docker service scale'
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Fix scale pointer value in service inspect -p
Before this change, pretty output shows `859533555312` instead of `1`.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Add missing newline
Signed-off-by: Arnaud Porterie (icecrime) <arnaud.porterie@docker.com>
Fix command not displaying in service ls.
Closes #396.

Signed-off-by: Drew Erny <drew.erny@docker.com>
New format for setting mounts on a service.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upate from PR review.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
fix rebase
Signed-off-by: Victor Vieux <vieux@docker.com>
@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Jun 14, 2016

Contributor

LGTM 💯

Contributor

icecrime commented Jun 14, 2016

LGTM 💯

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jun 14, 2016

Contributor

🎉 🎉 LGTM

Contributor

mavenugo commented Jun 14, 2016

🎉 🎉 LGTM

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Jun 14, 2016

Collaborator

LGTM 💯 this is very good!

Collaborator

vieux commented Jun 14, 2016

LGTM 💯 this is very good!

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Jun 14, 2016

Contributor

API reference will be completed in a followup 👍

Contributor

icecrime commented Jun 14, 2016

API reference will be completed in a followup 👍

@GordonTheTurtle GordonTheTurtle added dco/no and removed dco/no labels Jun 14, 2016

@icecrime icecrime merged commit 9646d17 into swarm Jun 14, 2016

1 of 8 checks passed

documentation started
Details
experimental Jenkins build Docker-PRs-experimental 19905 is running
Details
gccgo Jenkins build Docker-PRs-gccgo 6772 is running
Details
janky Jenkins build Docker-PRs 28699 is running
Details
userns Jenkins build Docker-PRs-userns 10861 is running
Details
win2lin Jenkins build Docker-PRs-Win2Lin 27285 is running
Details
windowsTP5 Jenkins build Docker-PRs-WoW-TP5 3190 is running
Details
docker/dco-signed All commits signed
Details

@vieux vieux deleted the swarm-backend branch Jul 21, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment