Create service with ports failed

[jimmyxian]

I try to create service with ports, but failed.
Cmd:

root@SZX1000041895:/var/log/upstart# swarmctl service create --name centos_service2 --ports 100:80 --image centos  --args sleep,1000
2920rtubulyxffpv56x1tibx4

swarmkit log:

root@SZX1000041895:/# swarmd -d /tmp/node-1 --listen-control-api /var/run/docker/cluster/docker-swarmd.sock --hostname node-1
Warning: Specifying a valid address with --listen-remote-api may be necessary for other managers to reach this one.
INFO[0000] 8a4424ceb6f3ddd3 became follower at term 0   
INFO[0000] newRaft 8a4424ceb6f3ddd3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] 
INFO[0000] 8a4424ceb6f3ddd3 became follower at term 1   
INFO[0000] 8a4424ceb6f3ddd3 is starting a new election at term 1 
INFO[0000] 8a4424ceb6f3ddd3 became candidate at term 2  
INFO[0000] 8a4424ceb6f3ddd3 received vote from 8a4424ceb6f3ddd3 at term 2 
INFO[0000] 8a4424ceb6f3ddd3 became leader at term 2     
INFO[0000] raft.node: 8a4424ceb6f3ddd3 elected leader 8a4424ceb6f3ddd3 at term 2 
INFO[0000] Listening for connections                     addr=[::]:4242 proto=tcp
INFO[0000] Listening for local connections               addr=/var/run/docker/cluster/docker-swarmd.sock proto=unix
INFO[0000] node is ready                                
INFO[0307] state changed                                 state.transition=ASSIGNED->ACCEPTED task.id=5fahk2hcjcy4a7itqfb5hlvo5
INFO[0307] state changed                                 state.desired=RUNNING state.transition=ASSIGNED->ACCEPTED task.id=5fahk2hcjcy4a7itqfb5hlvo5
INFO[0307] state changed                                 state.desired=RUNNING state.transition=ACCEPTED->PREPARING task.id=5fahk2hcjcy4a7itqfb5hlvo5
ERRO[0307] fatal task error                              error=Error response from daemon: datastore for scope "global" is not initialized task.id=5fahk2hcjcy4a7itqfb5hlvo5
INFO[0307] state changed                                 state.desired=RUNNING state.transition=PREPARING->REJECTED task.id=5fahk2hcjcy4a7itqfb5hlvo5
WARN[0307] task updates not yet supported                task.id=5fahk2hcjcy4a7itqfb5hlvo5
INFO[0307] state changed                                 state.transition=ASSIGNED->ACCEPTED task.id=e3ddznzxemghycngu8qvrn1rd
INFO[0307] state changed                                 state.desired=ACCEPTED state.transition=ASSIGNED->ACCEPTED task.id=e3ddznzxemghycngu8qvrn1rd
WARN[0312] task updates not yet supported                task.id=e3ddznzxemghycngu8qvrn1rd
INFO[0312] state changed                                 state.desired=RUNNING state.transition=ACCEPTED->PREPARING task.id=e3ddznzxemghycngu8qvrn1rd
ERRO[0312] fatal task error
......

Docker daemon log:

ERRO[549871] Handler for POST /networks/create returned error: datastore for scope "global" is not initialized  
ERRO[549871] Handler for GET /containers/centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5/json returned error: No such container: centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5 
ERRO[549871] Handler for GET /containers/centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5/json returned error: No such container: centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5 
ERRO[549871] Handler for GET /containers/centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5/json returned error: No such container: centos_service2.1.5fahk2hcjcy4a7itqfb5hlvo5 
ERRO[549876] Handler for POST /networks/create returned error: datastore for scope "global" is not initialized  
ERRO[549876] Handler for GET /containers/centos_service2.1.e3ddznzxemghycngu8qvrn1rd/json returned error: No such container: centos_service2.1.e3ddznzxemghycngu8qvrn1rd 
ERRO[549876] Handler for GET /containers/centos_service2.1.e3ddznzxemghycngu8qvrn1rd/json returned error: No such container: centos_service2.1.e3ddznzxemghycngu8qvrn1rd 
ERRO[549876] Handler for GET /containers/centos_service2.1.e3ddznzxemghycngu8qvrn1rd/json returned error: No such container: centos_service2.1.e3ddznzxemghycngu8qvrn1rd 
ERRO[549881] Handler for POST /networks/create returned error: datastore for scope "global" is not initialized
......

Environment: One node

root@SZX1000041895:/# swarmd --version
swarmd github.com/docker/swarmkit 5eac5de.m
root@SZX1000041895:/# docker version
Client:
 Version:      1.12.0-dev
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e94be2f
 Built:        Sun Jun  5 23:20:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0-dev
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e94be2f
 Built:        Sun Jun  5 23:20:17 2016
 OS/Arch:      linux/amd64

[aluzzardi]

/cc @mrjana @mavenugo

[hujie5592427]

me too

[mavenugo]

@aluzzardi the error clearly indicates that the daemon is not running in the newly introduced agent mode (moby/libnetwork#1149). Without that, daemon will expect KV-Store to be configured for any overlay network to be created. Since swarmkit creates ingress network using overlay driver for any --port configuration, it is expected to see such an error.

[aluzzardi]

@mavenugo So there's no way right now to use bare swarmkit with multi-host?

[Matsca09]

Same problem with ports, but different error and configuration:

Swarmd version:
bin/swarmd github.com/docker/swarmkit be7bd8d

Docker version:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

with Consul configured for KV storage for overlay networks.

Commands executed:

ubuntu@node1:~$ bin/swarmctl -s swarmd-node1/api.socket network ls
ID                         Name     Driver
--                         ----     ------
buio0qzw9691tcvcb0mwkrs5c  ingress  overlay
ubuntu@node1:~$ bin/swarmctl -s swarmd-node1/api.socket network create --driver overlay --name overlaynet-test
0u7ywlhgy6p6slpkbz9qezwph
ubuntu@node1:~$ bin/swarmctl -s swarmd-node1/api.socket network ls
ID                         Name             Driver
--                         ----             ------
0u7ywlhgy6p6slpkbz9qezwph  overlaynet-test  overlay
buio0qzw9691tcvcb0mwkrs5c  ingress          overlay
ubuntu@node1:~$ bin/swarmctl -s swarmd-node1/api.socket service create --name nginx-test --image nginx:alpine --ports web:80 --network overlaynet-test
dgkm5z2s1dad8f342q4ajrkid

service inspect:

ubuntu@node1:~$ bin/swarmctl -s swarmd-node1/api.socket service inspect nginx-test
ID                 : 6906juny21wubbean5tu021nd
Name               : nginx-test
Replicas           : 0/1
Template           
 Container         
  Image            : nginx:alpine
  Networks:         0u7ywlhgy6p6slpkbz9qezwph
Ports:
    - Name             = web
      Protocol         = TCP
      Port             = 80
      SwarmPort        = 30000

Task ID                      Service       Slot    Image           Desired State    Last State               Node
-------                      -------       ----    -----           -------------    ----------               ----
0ea67zok7d22hpj0ws65ns6gk    nginx-test    1       nginx:alpine    ACCEPTED         ACCEPTED 1 second ago    swarmd-node1

Networks listing in Docker:

NETWORK ID          NAME                DRIVER
1ead7b8f2923        bridge              bridge              
7170e3343efa        docker_gwbridge     bridge              
54027a17e80f        host                host                
49cad312fc5e        ingress             overlay             
c5523c29f3e4        none                null                
6d84b23cda33        overlaynet-test     overlay 

swarmd output:

ubuntu@node1:~$ bin/swarmd --hostname swarmd-node1 --listen-remote-api 172.25.0.17:4242 -d /home/ubuntu/swarmd-node1 --listen-control-api /home/ubuntu/swarmd-node1/api.socket
INFO[0000] 36a0ef7789f4d7ae became follower at term 0   
INFO[0000] newRaft 36a0ef7789f4d7ae [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] 
INFO[0000] 36a0ef7789f4d7ae became follower at term 1   
INFO[0000] 36a0ef7789f4d7ae is starting a new election at term 1 
INFO[0000] 36a0ef7789f4d7ae became candidate at term 2  
INFO[0000] 36a0ef7789f4d7ae received vote from 36a0ef7789f4d7ae at term 2 
INFO[0000] 36a0ef7789f4d7ae became leader at term 2     
INFO[0000] raft.node: 36a0ef7789f4d7ae elected leader 36a0ef7789f4d7ae at term 2 
INFO[0000] Listening for connections                     addr=172.25.0.17:4242 proto=tcp
INFO[0000] Listening for local connections               addr=/home/ubuntu/swarmd-node1/api.socket proto=unix
INFO[0000] node is ready                                
ERRO[0094] fatal task error                              error=Error response from daemon: Container cannot be connected to network endpoints: ingress, overlaynet-test module=taskmanager task.id=9nx24mzqydg0e3q4c5bq4u9yt
WARN[0094] task updates not yet supported                module=taskmanager task.id=9nx24mzqydg0e3q4c5bq4u9yt
WARN[0099] task updates not yet supported                module=taskmanager task.id=648jln7cchrrh54wknjvcv8b6

docker logs:

Jun 16 17:09:07 node1 docker[30832]: time="2016-06-16T17:09:07.669060657Z" level=error msg="Handler for POST /containers/create returned error: Container cannot be connected to network endpo
ints: ingress, overlaynet-test"
Jun 16 17:09:07 node1 docker[30832]: time="2016-06-16T17:09:07.671831160Z" level=error msg="Handler for GET /containers/nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt/json returned error: No such container: nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt"
Jun 16 17:09:07 node1 docker[30832]: time="2016-06-16T17:09:07.674202666Z" level=error msg="Handler for GET /containers/nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt/json returned error: No such container: nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt"
Jun 16 17:09:07 node1 docker[30832]: time="2016-06-16T17:09:07.710721928Z" level=error msg="Handler for GET /containers/nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt/json returned error: No such container: nginx-test.1.9nx24mzqydg0e3q4c5bq4u9yt"

Same error if I run swarmd/swarmctl as root.

Without specifying any port, the container is created and connected to the "overlaynet-test" network (tried with swarmkit and manually with "docker run").

[jinuxstyle]

@Matsca09 I met the same problem. Even the docker daemon started in agent mode, the swarmkit service still can not run successfully if it's given a --port option. It looks swarmkit will try to create the default "ingress" network even user specified a different network by --network option. Unfortunately the docker daemon doesn't allow creating network named "ingress". So the swarmkit would fail at creating containers. Following code snippet indicates the reason underneath.

docker/daemon/network.go:createNetwork()

if runconfig.IsPreDefinedNetwork(create.Name) && !agent {
    err := fmt.Errorf("%s is a pre-defined network and cannot be created", create.Name)
    return nil, errors.NewRequestForbiddenError(err)
}

docker/runconfig/hostconfig_unix.go

// IsPreDefinedNetwork indicates if a network is predefined by the daemon
func IsPreDefinedNetwork(network string) bool {
    n := container.NetworkMode(network)
    return n.IsBridge() || n.IsHost() || n.IsNone() || n.IsDefault() || network == "ingress"
}

From swarmkit side, the default network name is defined in swarmkit/manager/allocator/network.go
ingressNetworkName = "ingress"

@aluzzardi, should we change the default network name "ingress" to avoid the problem?

[olljanat]

@jimmyxian I'm going through old issues here.

Is this one still valid or can we close it?