New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support swarm-mode services with node-local networks #32981

Merged
merged 7 commits into from May 18, 2017

Conversation

@aboch
Contributor

aboch commented May 3, 2017

This PR makes changes to support creating swarm networks with macvlan driver, or with any other local datascope driver capable of creating multihost networks. Edit: Now also over the default bridge and host networks.

See #27082

If the network driver is also capable of creating networks which provide connectivity across hosts (as macvlan driver for example), then the service discovery will be provided as it is done for global datascope drivers like overlay.

In order to achieve that, it adds three new flags to the network create API: multihost scope=swarm, config-only, config-from.

While a global datascope driver network is by default created as a swarm network (see overlay nw as example), we need an option to make a local scope driver network creation a swarm network creation.
multihost scope=swarm option is for that. This is needed to ensure both regular and swarm networks from the same driver can coexist.

Edit: Note: I am now thinking this flag should be renamed multihost --> scope=swarm as its actual semantic is "I want this network to be a swarm visible network"

For local datascope driver networks, resource allocation has to happen on the host, it cannot happen in the swarm manager, because the resources are tied to the underlay network and/or (as for macvlan) to the host network interfaces, so they are different in each swarm node. Once the dynamic swarm network creation request is sent to libnetwork, libnetwork on each swarm node needs to know from where to pick the local scope (therefore node specific) configuration for the network.

This is fine for simple local scope single host driver networks, which do not require specific configurations and are fine with the default IPAM resources. But for local scope, multihost capable driver networks, there is a need to provide a global identification of the docker network across the swarm nodes, being a multihost network entity, besides the need for host specific network configurations, like the parent interface for macvlan networks for example.

In order to solve this, user is given the ability to create config-only networks which contain network specific configurations on each host. This kind of network is saved to store but no allocation or plumbing is attempted for it. The config-from option comes to play when the client creates the swarm network and wants to specify which configuration network will provide the node local network specific configurations.

Now user can create a, say, macvlan swarm network toward the swarm manager specifying that the configuration information comes from another network which will be present on the nodes, with

docker network create -d macvlan --scope=swarm --config-from mv-config swarm-mv-nw

after he has created the config networks on the swarm nodes with:

node1# docker network create --config-only --subnet 192.168.10.0/24 --gateway 192.168.10.1 --opt parent=enp8s0 --macvlan_mode=bridge mv-conf

node2# docker network create --config-only --subnet 192.168.20.0/24 --gateway 192.168.20.1 --opt parent=eth1 --macvlan_mode=bridge mv-conf

Note:
command reference and swagger changes, integration tests will be added as design review proceeds.

Note: If a node-local multihost capable driver is capable to autonomously allocate its resources on each node, then there is no need for the user to create a config-only network and pass it during the swarm network creation as config-from

Closes #27082

  • Libnetwork vendoring:
    Fixes #29184
    Fixes #23987
    Related to#24848
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 3, 2017

Member

@aboch build is failing;

01:49:07 Building: bundles/17.06.0-dev/binary-daemon/dockerd-17.06.0-dev
01:49:54 # github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/allocator/networkallocator
01:49:54 vendor/github.com/docker/swarmkit/manager/allocator/networkallocator/networkallocator.go:142: undefined: dName
Member

thaJeztah commented May 3, 2017

@aboch build is failing;

01:49:07 Building: bundles/17.06.0-dev/binary-daemon/dockerd-17.06.0-dev
01:49:54 # github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/allocator/networkallocator
01:49:54 vendor/github.com/docker/swarmkit/manager/allocator/networkallocator/networkallocator.go:142: undefined: dName
@rreinurm

This comment has been minimized.

Show comment
Hide comment
@rreinurm

rreinurm May 3, 2017

If I understand the design correctly. In order to attach containers from swarm service to default bridge network I should create multihost network like this?

docker network create -d bridge --multihost --config-from bridge swarm-bridge-nw

Assuming that bridge network has attribute config-only set to true?

rreinurm commented May 3, 2017

If I understand the design correctly. In order to attach containers from swarm service to default bridge network I should create multihost network like this?

docker network create -d bridge --multihost --config-from bridge swarm-bridge-nw

Assuming that bridge network has attribute config-only set to true?

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 3, 2017

Contributor

@rreinurm that is correct if the original network is created with a --config-only flag as bridge. But in this case, since the default docker0 bridge is not created with --config-only, then it wont work as you described.

@aboch Can you pls update the PR title from macvlan -> node-local ?
Also can you pls refer to the original proposal #27082 from your description ?

Contributor

mavenugo commented May 3, 2017

@rreinurm that is correct if the original network is created with a --config-only flag as bridge. But in this case, since the default docker0 bridge is not created with --config-only, then it wont work as you described.

@aboch Can you pls update the PR title from macvlan -> node-local ?
Also can you pls refer to the original proposal #27082 from your description ?

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 3, 2017

Contributor

@aboch can you pls give more details on why we need --config-only and --config-from options in order to make use of node-local networks ?
If I remember correct, it was because of the need to have the same network-id on all the hosts and hence the actual swarm network (that uses the --config-from option) must be created at the swarm manager so that it can provide consistent network-id across the workers.

In addition to these user-defined networks, how can we make use of existing default networks (such as docker0 and host networks) ?
Is there a way we can fix docker service create to make use of bridge and host network ?

Contributor

mavenugo commented May 3, 2017

@aboch can you pls give more details on why we need --config-only and --config-from options in order to make use of node-local networks ?
If I remember correct, it was because of the need to have the same network-id on all the hosts and hence the actual swarm network (that uses the --config-from option) must be created at the swarm manager so that it can provide consistent network-id across the workers.

In addition to these user-defined networks, how can we make use of existing default networks (such as docker0 and host networks) ?
Is there a way we can fix docker service create to make use of bridge and host network ?

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch May 3, 2017

Contributor

I updated the description, to make it clear that the multihost (but would be better swarm) flag is for creating a swarm usable network with any local datascope network driver.

The fact that the network driver is multihost capable is an internal detail between the network driver and libnetwork core and it is handled there transparently to the user.

@rreinurm The reason why you would not be able to run a service on the default bridge network is because that network was not created with the multihost (read swarm) flag.

If you do not have any specific nw configurations for the bridge network (read you do not need the --config-only and --config-from flags) where you want to run your tasks, then you can just create a bridge network with the --swarm flag. That should be all what is needed.

Contributor

aboch commented May 3, 2017

I updated the description, to make it clear that the multihost (but would be better swarm) flag is for creating a swarm usable network with any local datascope network driver.

The fact that the network driver is multihost capable is an internal detail between the network driver and libnetwork core and it is handled there transparently to the user.

@rreinurm The reason why you would not be able to run a service on the default bridge network is because that network was not created with the multihost (read swarm) flag.

If you do not have any specific nw configurations for the bridge network (read you do not need the --config-only and --config-from flags) where you want to run your tasks, then you can just create a bridge network with the --swarm flag. That should be all what is needed.

}
if err := json.Unmarshal(ba, &driverOptions); err != nil {
return fmt.Errorf("failed to validate network configuration: %v", err)
}

This comment has been minimized.

@aaronlehmann

aaronlehmann May 3, 2017

Contributor

Trying to understand this. We marshal the data to JSON, and then unmarshal it back? I guess the idea is to convert map[string]interface{} to map[string]string, but I think a simple loop over the map would be a more straightforward way.

@aaronlehmann

aaronlehmann May 3, 2017

Contributor

Trying to understand this. We marshal the data to JSON, and then unmarshal it back? I guess the idea is to convert map[string]interface{} to map[string]string, but I think a simple loop over the map would be a more straightforward way.

This comment has been minimized.

@aboch

aboch May 4, 2017

Contributor

The problem is that opts may be of type map[string]interface{} or map[string]string. Because we initially used the options.Generic for the driver options, the go json conversion of these types has been a continuous headache.

The Marshal & Unmarshal pattern has proven to be the simplest way to go around the issue. You can see it used in several places in libentwork.

@aboch

aboch May 4, 2017

Contributor

The problem is that opts may be of type map[string]interface{} or map[string]string. Because we initially used the options.Generic for the driver options, the go json conversion of these types has been a continuous headache.

The Marshal & Unmarshal pattern has proven to be the simplest way to go around the issue. You can see it used in several places in libentwork.

// ConfigFrom indicates that the network specific configuration
// for this network will be provided via another network, locally
// on the node where this network is being plumbed.
string config_from = 8;

This comment has been minimized.

@aaronlehmann

aaronlehmann May 3, 2017

Contributor

cc @ijc25

@aaronlehmann

aaronlehmann May 3, 2017

Contributor

cc @ijc25

@aboch aboch changed the title from Add support for creating macvlan swarm networks to Support swarm-mode services with node-local networks May 4, 2017

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 6, 2017

Contributor

@aboch I tried the PR and the functionalities are working as advertised. Few overall comments -

  • cli has been moved out of this repo and into docker/cli project. Pls move the CLI portions and rebase this PR
  • In order to best use this feature, it must also support the default bridge and host networks. Once we have these networks usable in service create, then it will unlock a lot of potential and accelerate the adoption of service concept.
Contributor

mavenugo commented May 6, 2017

@aboch I tried the PR and the functionalities are working as advertised. Few overall comments -

  • cli has been moved out of this repo and into docker/cli project. Pls move the CLI portions and rebase this PR
  • In order to best use this feature, it must also support the default bridge and host networks. Once we have these networks usable in service create, then it will unlock a lot of potential and accelerate the adoption of service concept.
@eyz

This comment has been minimized.

Show comment
Hide comment
@eyz

eyz May 7, 2017

@aboch Does this work with ipvlan as well? The following was tested against a CentOS 7 build of aboch/docker@be38dba -

[root@centos ~]# docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 -o ipvlan_mode=l2 -o parent=ens192 ipvlan_l2
6e4ff937776fe09324be620620df706fdac96aa900c3b7d3ac26243d37658bb3

[root@centos ~]# docker network create -d ipvlan --scope=swarm --config-from ipvlan_l2 swarm_ipvlan_l2
Error response from daemon: rpc error: code = 3 desc = error during lookup of plugin ipvlan

Thank you for working on this integration. If merged, it will help us adopt Swarm, as we are using ipvlan l2 currently.

Also, can global-scoped IPAM be used with this, or is that only applicable to overlay for now?

I've confirmed the locally-scoped IPAM works fine via macvlan, which is still a pretty big win -

docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.64/26 -o parent=ens192 test_macvlan_bridge # node 1
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.128/26 -o parent=ens192 test_macvlan_bridge # node 2
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.192/26 -o parent=ens192 test_macvlan_bridge # node 3

eyz commented May 7, 2017

@aboch Does this work with ipvlan as well? The following was tested against a CentOS 7 build of aboch/docker@be38dba -

[root@centos ~]# docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 -o ipvlan_mode=l2 -o parent=ens192 ipvlan_l2
6e4ff937776fe09324be620620df706fdac96aa900c3b7d3ac26243d37658bb3

[root@centos ~]# docker network create -d ipvlan --scope=swarm --config-from ipvlan_l2 swarm_ipvlan_l2
Error response from daemon: rpc error: code = 3 desc = error during lookup of plugin ipvlan

Thank you for working on this integration. If merged, it will help us adopt Swarm, as we are using ipvlan l2 currently.

Also, can global-scoped IPAM be used with this, or is that only applicable to overlay for now?

I've confirmed the locally-scoped IPAM works fine via macvlan, which is still a pretty big win -

docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.64/26 -o parent=ens192 test_macvlan_bridge # node 1
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.128/26 -o parent=ens192 test_macvlan_bridge # node 2
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.192/26 -o parent=ens192 test_macvlan_bridge # node 3

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch May 8, 2017

Contributor

Thanks @eyz, yes I did not add it because it is an experimental driver. I will add the support for it. so that user who is fine to run the daemon with the experimental flag can make use of it.

Contributor

aboch commented May 8, 2017

Thanks @eyz, yes I did not add it because it is an experimental driver. I will add the support for it. so that user who is fine to run the daemon with the experimental flag can make use of it.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 9, 2017

Contributor

@aboch I understand this is still in design review. But it will help if you can open the PRs against libnetwork and swarmkit individually so that it makes it easier to review the components while the moby side glue can be reviewed in this PR.

From design standpoint, am glad that this PR addresses the overall theme. Just a few additional changes will make it even better.

  1. Support for moby default networks bridge and host
  2. IMO, It is better to have config-only networks created with null driver (which will aid with downgrades).
Contributor

mavenugo commented May 9, 2017

@aboch I understand this is still in design review. But it will help if you can open the PRs against libnetwork and swarmkit individually so that it makes it easier to review the components while the moby side glue can be reviewed in this PR.

From design standpoint, am glad that this PR addresses the overall theme. Just a few additional changes will make it even better.

  1. Support for moby default networks bridge and host
  2. IMO, It is better to have config-only networks created with null driver (which will aid with downgrades).
@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch May 9, 2017

Contributor

@mavenugo

IMO, It is better to have config-only networks created with null driver (which will aid with downgrades).

This is already taken care

Support for moby default networks bridge and host

The requirement this PR addresses does did not include running services on host and default bridge network. Honestly it is not easy to fit this extra requirement in here. I am afraid we'll need to add some special case code to address that. IMO, we should address it in a separate PR, once this one is in.

Contributor

aboch commented May 9, 2017

@mavenugo

IMO, It is better to have config-only networks created with null driver (which will aid with downgrades).

This is already taken care

Support for moby default networks bridge and host

The requirement this PR addresses does did not include running services on host and default bridge network. Honestly it is not easy to fit this extra requirement in here. I am afraid we'll need to add some special case code to address that. IMO, we should address it in a separate PR, once this one is in.

@@ -396,13 +396,15 @@ type NetworkResource struct {
Name string // Name is the requested name of the network
ID string `json:"Id"` // ID uniquely identifies a network on a single machine
Created time.Time // Created is the time the network created
Scope string // Scope describes the level at which the network exists (e.g. `global` for cluster-wide or `local` for machine level)
Scope string // Scope describes the level at which the network exists (e.g. `swarm` for cluster-wide or `local` for machine level)

This comment has been minimized.

@dongluochen

dongluochen May 10, 2017

Contributor

I prefer global to swarm. swarm itself doesn't have clear meaning as a network scope. swarm could be aware of a network on a local node.

@dongluochen

dongluochen May 10, 2017

Contributor

I prefer global to swarm. swarm itself doesn't have clear meaning as a network scope. swarm could be aware of a network on a local node.

This comment has been minimized.

@aboch

aboch May 12, 2017

Contributor

We discussed this internally, and the reason we opted for swarm is because it is easy to associate to the swarm feature. Also in case of single host networks promoted to swarm scope, global would not be correct because the network does is not a multihost network. It just indicates the network is usable within swarm.

@aboch

aboch May 12, 2017

Contributor

We discussed this internally, and the reason we opted for swarm is because it is easy to associate to the swarm feature. Also in case of single host networks promoted to swarm scope, global would not be correct because the network does is not a multihost network. It just indicates the network is usable within swarm.

This comment has been minimized.

@mavenugo

mavenugo May 13, 2017

Contributor

@dongluochen We already are using the swarm scope terminology in docker network ls even though the network is backed by a global-scoped driver. But with this PR, we are also supporting local-scoped drivers, but it can be used in swarm-mode. With the current used terminology and the scope of using the network at a swarm level, I think swarm scope indicates it appropriately. WDYT ?

@mavenugo

mavenugo May 13, 2017

Contributor

@dongluochen We already are using the swarm scope terminology in docker network ls even though the network is backed by a global-scoped driver. But with this PR, we are also supporting local-scoped drivers, but it can be used in swarm-mode. With the current used terminology and the scope of using the network at a swarm level, I think swarm scope indicates it appropriately. WDYT ?

Show outdated Hide outdated api/types/types.go Outdated
@dongluochen

This comment has been minimized.

Show comment
Hide comment
@dongluochen

dongluochen May 10, 2017

Contributor

There are issues asking for local load balancing support. Tasks from service A running on a node should access service B instances running on the same node. Does node-local network enable such scenarios?

Contributor

dongluochen commented May 10, 2017

There are issues asking for local load balancing support. Tasks from service A running on a node should access service B instances running on the same node. Does node-local network enable such scenarios?

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch May 10, 2017

Contributor

@dongluochen This PR addresses the ability to launch services or containers on any node-local network,
read bridge, macvlan, ipvlan. The service discovery on the same network should be available, regardless the network is single or multihost.

Contributor

aboch commented May 10, 2017

@dongluochen This PR addresses the ability to launch services or containers on any node-local network,
read bridge, macvlan, ipvlan. The service discovery on the same network should be available, regardless the network is single or multihost.

@eyz

This comment has been minimized.

Show comment
Hide comment
@eyz

eyz May 10, 2017

There is probably a better place to ask this question, and so I apologize ahead of time, but, possibly related to @dongluochen's question, and something that has not been clear to me: is the Docker Swarm IPVS L4 VIP-based load balancing specific to the overlay network driver, or is active on any globally-scoped network driver, such as this implementation?

I know multi-record round robin DNS queries are working fine with this implementation, in my testing. However, the only references to the IPVS VIP-based load balancing I've found mention its use with overlay, such as shown here - https://success.docker.com/@api/deki/files/203/driver-comparison.png

eyz commented May 10, 2017

There is probably a better place to ask this question, and so I apologize ahead of time, but, possibly related to @dongluochen's question, and something that has not been clear to me: is the Docker Swarm IPVS L4 VIP-based load balancing specific to the overlay network driver, or is active on any globally-scoped network driver, such as this implementation?

I know multi-record round robin DNS queries are working fine with this implementation, in my testing. However, the only references to the IPVS VIP-based load balancing I've found mention its use with overlay, such as shown here - https://success.docker.com/@api/deki/files/203/driver-comparison.png

@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch May 11, 2017

Contributor

@mavenugo

As requested, I added the support for running docker services on predefined networks (bridge and host).

Please give it a try.

$ docker service create --name host-srv --network host busybox top
jsvdzctxv8g67wogshhqcx3e3
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
$ docker service ps host-srv 
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
sg1ge61vgf3z        host-srv.1          busybox:latest      aboch-XPS           Running             Running 7 seconds ago                       
$ docker exec host-srv.1.sg1ge61vgf3ziab92d4ausp6f ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:42:06:B6:BF:6B  
          inet addr:172.18.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:6ff:feb6:bf6b/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:125730 errors:0 dropped:0 overruns:0 frame:0
          TX packets:316809 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6859706 (6.5 MiB)  TX bytes:439978635 (419.5 MiB)

$ docker network inspect host
[
    {
        "Name": "host",
        "Id": "a6e94f7c2e6821ea53cfadf781c04943f37f750c750fd35ff48d3c14ee19ee8a",
        "Created": "2017-05-11T11:24:37.0910853-07:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": "",
        "ConfigOnly": false,
        "Containers": {
            "473e0fa2dff1fb377f89f00a20b11d2d983664332fdd1146baee772423673b06": {
                "Name": "host-srv.1.sg1ge61vgf3ziab92d4ausp6f",
                "EndpointID": "75240cd166fd882ae4c3382d1fc28c48f279166b05af18cdd6ba96b49e8e4039",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
$ 
Contributor

aboch commented May 11, 2017

@mavenugo

As requested, I added the support for running docker services on predefined networks (bridge and host).

Please give it a try.

$ docker service create --name host-srv --network host busybox top
jsvdzctxv8g67wogshhqcx3e3
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
$ docker service ps host-srv 
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
sg1ge61vgf3z        host-srv.1          busybox:latest      aboch-XPS           Running             Running 7 seconds ago                       
$ docker exec host-srv.1.sg1ge61vgf3ziab92d4ausp6f ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:42:06:B6:BF:6B  
          inet addr:172.18.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:6ff:feb6:bf6b/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:125730 errors:0 dropped:0 overruns:0 frame:0
          TX packets:316809 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6859706 (6.5 MiB)  TX bytes:439978635 (419.5 MiB)

$ docker network inspect host
[
    {
        "Name": "host",
        "Id": "a6e94f7c2e6821ea53cfadf781c04943f37f750c750fd35ff48d3c14ee19ee8a",
        "Created": "2017-05-11T11:24:37.0910853-07:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": "",
        "ConfigOnly": false,
        "Containers": {
            "473e0fa2dff1fb377f89f00a20b11d2d983664332fdd1146baee772423673b06": {
                "Name": "host-srv.1.sg1ge61vgf3ziab92d4ausp6f",
                "EndpointID": "75240cd166fd882ae4c3382d1fc28c48f279166b05af18cdd6ba96b49e8e4039",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
$ 
@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 18, 2017

Contributor

windows-rs1 CI (https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14121/console) is stuck at the starting point and also Jenkins is acting up and very slow to even get the correct status.

@aboch confirmed that windows-rs1 CI was green in his previous attempts (https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14106/consoleFull)

With all other CIs green, I will go ahead with the merge to unblock subsequent PRs.

==== EDIT =====

Windows-RS1 Re-run successful : https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14131/console

Contributor

mavenugo commented May 18, 2017

windows-rs1 CI (https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14121/console) is stuck at the starting point and also Jenkins is acting up and very slow to even get the correct status.

@aboch confirmed that windows-rs1 CI was green in his previous attempts (https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14106/consoleFull)

With all other CIs green, I will go ahead with the merge to unblock subsequent PRs.

==== EDIT =====

Windows-RS1 Re-run successful : https://jenkins.dockerproject.org/job/Docker-PRs-WoW-RS1/14131/console

@mavenugo mavenugo merged commit ca4b3c6 into moby:master May 18, 2017

6 of 7 checks passed

windowsRS1 Jenkins build Docker-PRs-WoW-RS1 14131 is running
Details
dco-signed All commits are signed
experimental Jenkins build Docker-PRs-experimental 34280 has succeeded
Details
janky Jenkins build Docker-PRs 42881 has succeeded
Details
powerpc Jenkins build Docker-PRs-powerpc 3265 has succeeded
Details
vendor Jenkins build Docker-PRs-vendor 3437 has succeeded
Details
z Jenkins build Docker-PRs-s390x 2984 has succeeded
Details
@eyz

This comment has been minimized.

Show comment
Hide comment
@eyz

eyz May 18, 2017

Hi all. First off: LGTM 👍 🎉

I had great success testing this PR #32981 with ipvlan on CentOS 7.x. The manpages still aren't building in the Makefile, but otherwise it is working great -

  • checked out of github.com/moby/moby 45c6f42 (was HEAD of master at the time of my checkout) into my go/src/github.com/moby, as go/src/github.com/moby/moby
  • removed all contrib/builder deb and rpm subfolders except contrib/builder/rpm/amd64/centos-7
isaac.rodman@ubuntu:/home/isaac/go/src/github.com/moby/moby# find contrib/builder/
contrib/builder/
contrib/builder/rpm
contrib/builder/rpm/amd64
contrib/builder/rpm/amd64/build.sh
contrib/builder/rpm/amd64/README.md
contrib/builder/rpm/amd64/generate.sh
contrib/builder/rpm/amd64/centos-7
contrib/builder/rpm/amd64/centos-7/Dockerfile
  • commented manpages in Makefile
#manpages: ## Generate man pages from go source and markdown
#       docker build ${DOCKER_BUILD_ARGS} -t docker-manpage-dev -f "man/$(DOCKERFILE)" ./man
#       docker run --rm \
#               -v $(PWD):/go/src/github.com/docker/docker/ \
#               docker-manpage-dev
  • attempted to make rpm (prior to further required modifications)
make rpm
  • waited for fail due to containerd tar not completing (below)
---> Making bundle: build-rpm (in bundles/17.06.0-dev/build-rpm)
Using test binary docker
+++ /etc/init.d/apparmor start
Starting AppArmor profiles:Warning from stdin (line 1): /sbin/apparmor_parser: cannot use or update cache, disable, or force-complain via stdin
.
INFO: Waiting for daemon to start...
+++ exec dockerd --debug --host unix:///go/src/github.com/docker/docker/bundles/17.06.0-dev/build-rpm/docker.sock --storage-driver btrfs --pidfile bundles/17.06.0-dev/build-rpm/docker.pid --userland-proxy=true
.
make: Nothing to be done for 'manpages'.
++ docker build -t dockercore/builder-rpm:centos-7 contrib/builder/rpm/amd64/centos-7/
Sending build context to Docker daemon   2.56kB
Step 1/10 : FROM centos:7
7: Pulling from library/centos
343b09361036: Pulling fs layer
343b09361036: Verifying Checksum
343b09361036: Download complete
343b09361036: Pull complete
Digest: sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077
Status: Downloaded newer image for centos:7
 ---> 8140d0c64310

[... elided ...]

Step 10/18 : WORKDIR /root/rpmbuild/SPECS
 ---> 941f5e39f6b8
Removing intermediate container 9065ac639458
Step 11/18 : RUN tar --exclude .git -r -C /usr/src -f /root/rpmbuild/SOURCES/docker-engine.tar docker-engine
 ---> Running in b917994182b2
 ---> e1657a479ebe
Removing intermediate container b917994182b2
Step 12/18 : RUN tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
 ---> Running in c2d757160a23
tar: containerd: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
The command '/bin/sh -c tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd' returned a non-zero code: 2
---> Making bundle: .integration-daemon-stop (in bundles/17.06.0-dev/build-rpm)
+++++ cat bundles/17.06.0-dev/build-rpm/docker.pid
++++ kill 3258
++++ /etc/init.d/apparmor stop
Clearing AppArmor profiles cache:.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
Makefile:144: recipe for target 'rpm' failed
make: *** [rpm] Error 1
  • commented these lines in bundles/17.06.0-dev/build-rpm/centos-7/Dockerfile.build
#RUN tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
#RUN tar --exclude .git -r -C /go/src/github.com/docker/libnetwork/cmd -f /root/rpmbuild/SOURCES/docker-engine.tar proxy
#RUN tar --exclude .git -r -C /go/src/github.com/opencontainers -f /root/rpmbuild/SOURCES/docker-engine.tar runc
#RUN tar --exclude .git -r -C /go/ -f /root/rpmbuild/SOURCES/docker-engine.tar tini
#RUN gzip /root/rpmbuild/SOURCES/docker-engine.tar
#RUN { cat /usr/src/docker-engine/contrib/builder/rpm/amd64/changelog; } >> docker-engine.spec && tail >&2 docker-engine.spec
#RUN rpmbuild -ba --define '_gitcommit 4874e05-unsupported' --define '_release 0.0.20170518.023124.git4874e05' --define '_version 17.06.0' --define '_origversion 17.06.0-dev' --define '_experimental 0' docker-engine.spec

^ removed extra whitespace on rpmbuild line after generated, to paste easier later

  • launched partial container (built without the lines we commented, above; we will run those manually after some modifications to the docker-engine.spec, below)
docker build -t docker-temp/build-rpm:centos-7 -f bundles/17.06.0-dev/build-rpm/centos-7/Dockerfile.build .
docker run -ti docker-temp/build-rpm:centos-7 bash
  • ensure you are in the /root/rpmbuild/SPECS folder inside the newly-built docker-temp/build-rpm:centos-7 container
cd /root/rpmbuild/SPECS
  • commented manpages lines in docker-engine.spec (first section)
# install manpages
#install -d %{buildroot}%{_mandir}/man1
#install -p -m 644 man/man1/*.1 $RPM_BUILD_ROOT/%{_mandir}/man1
#install -d %{buildroot}%{_mandir}/man5
#install -p -m 644 man/man5/*.5 $RPM_BUILD_ROOT/%{_mandir}/man5
#install -d %{buildroot}%{_mandir}/man8
#install -p -m 644 man/man8/*.8 $RPM_BUILD_ROOT/%{_mandir}/man8
  • commented docker CLI package in docker-engine.spec (second section; built from docker/cli master instead)
%files
%doc AUTHORS CHANGELOG.md CONTRIBUTING.md LICENSE MAINTAINERS NOTICE README.md
#/%{_bindir}/docker
  • commented manpages lines in docker-engine.spec (third section)
%doc
#/%{_mandir}/man1/*
#/%{_mandir}/man5/*
#/%{_mandir}/man8/*
  • manually ran commented lines from Dockerfile (except containerd and changelog append)
# tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
tar --exclude .git -r -C /go/src/github.com/docker/libnetwork/cmd -f /root/rpmbuild/SOURCES/docker-engine.tar proxy
tar --exclude .git -r -C /go/src/github.com/opencontainers -f /root/rpmbuild/SOURCES/docker-engine.tar runc
tar --exclude .git -r -C /go/ -f /root/rpmbuild/SOURCES/docker-engine.tar tini
gzip /root/rpmbuild/SOURCES/docker-engine.tar
# { cat /usr/src/docker-engine/contrib/builder/rpm/amd64/changelog; } >> docker-engine.spec && tail >&2 docker-engine.spec
rpmbuild -ba --define '_gitcommit 4874e05-unsupported' --define '_release 0.0.20170518.023124.git4874e05' --define '_version 17.06.0' --define '_origversion 17.06.0-dev' --define '_experimental 0' docker-engine.spec
  • observed that CentOS 7 RPMs were built
[root@63168ae7b4ef SPECS]# find /root/rpmbuild/RPMS/x86_64/ -type f
/root/rpmbuild/RPMS/x86_64/docker-engine-17.06.0-0.0.20170518.023124.git4874e05.el7.centos.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/docker-engine-debuginfo-17.06.0-0.0.20170518.023124.git4874e05.el7.centos.x86_64.rpm
  • copied built RPMs to nodes 01 thru 03
  • installed RPMs (nodes 01 thru 03, on CentOS 7.3)
yum -y install container-selinux
rpm -i docker-*.rpm
  • centos7swarmtest01 (Swarm master)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.64/26 -o parent=ens192 test_config_only # node 1
  • centos7swarmtest02 (Swarm worker)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.128/26 -o parent=ens192 test_config_only # node 2
  • centos7swarmtest03 (Swarm worker)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.192/26 -o parent=ens192 test_config_only # node 3
  • centos7swarmtest01 (Swarm master)
docker network create -d ipvlan --scope=swarm --config-from test_config_only --attachable swarm_test
docker deploy -c /root/docker-compose-v3-test.yml voting
version: "3"

services:

  redis:
    image: redis:3.2-alpine
    ports:
      - "6379"
    networks:
      - votingnet
    deploy:
      placement:
        constraints: [node.role == manager]

  db:
    image: postgres:9.4
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - votingnet
    deploy:
      placement:
        constraints: [node.role == manager]

  voting-app:
    image: gaiadocker/example-voting-app-vote:good
    ports:
      - 5000:80
    networks:
      - votingnet
    depends_on:
      - redis
    deploy:
      mode: replicated
      replicas: 2
      labels: [APP=VOTING]
      placement:
        constraints: [node.role == worker]

  result-app:
    image: gaiadocker/example-voting-app-result:latest
    ports:
      - 5001:80
    networks:
      - votingnet
    depends_on:
      - db

  worker:
    image: gaiadocker/example-voting-app-worker:latest
    networks:
      votingnet:
        aliases:
          - workers
    depends_on:
      - db
      - redis
    # service deployment
    deploy:
      mode: replicated
      replicas: 2
      labels: [APP=VOTING]
      # service resource management
      resources:
        # Hard limit - Docker does not allow to allocate more
        limits:
          cpus: '0.25'
          memory: 512M
        # Soft limit - Docker makes best effort to return to it
        reservations:
          cpus: '0.25'
          memory: 256M
      # service restart policy
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      # service update configuration
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: continue
        monitor: 60s
        max_failure_ratio: 0.3
      # placement constraint - in this case on 'worker' nodes only
      placement:
        constraints: [node.role == worker]

networks:
    votingnet:
      external:
        name: swarm_test

volumes:
  db-data:
  • verified stack on centos7swarmtest01 (Swarm master)
[root@centos7swarmtest01 ~]# docker service ps --no-trunc voting
ID                          NAME                  IMAGE                                            NODE                DESIRED STATE       CURRENT STATE                     ERROR                                                             PORTS
51ebcxozsf8clwd5c3gj10l6c   voting_worker.1       gaiadocker/example-voting-app-worker:latest      centos7swarmtest02   Running             Running 29 minutes ago                                                                              
7ubvm4znlk2lls2sbw2orri7b   voting_result-app.1   gaiadocker/example-voting-app-result:latest      centos7swarmtest02   Running             Running 29 minutes ago                                                                              
znz47wst93kfic0lxlgdteoz6   voting_voting-app.1   gaiadocker/example-voting-app-vote:good          centos7swarmtest03   Running             Running 30 minutes ago                                                                              
r2f8r28z6nxe5bbk5t7wl2rbc   voting_db.1           postgres:9.4                                     centos7swarmtest01   Running             Running 29 minutes ago                                                                              
7xe7da400oycj3w0qs48d8xbk   voting_redis.1        redis:3.2-alpine                                 centos7swarmtest01   Running             Running 30 minutes ago                                                                              
zg97amwo70q800bgs11pe2h1p   voting_worker.2       gaiadocker/example-voting-app-worker:latest      centos7swarmtest03   Running             Running 29 minutes ago                                                                              
xeh62taucdw6uso3nrebogfp6   voting_voting-app.2   gaiadocker/example-voting-app-vote:good          centos7swarmtest02   Running             Running 29 minutes ago                   
  • verified MACs and IPs bound to centos7swarmtest01 (Swarm master using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest01 ~]# MAC='4f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
Containers -
4e50f691d32c:    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.65",
3bbc02a6a682:    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.6",
                    "IPAddress": "10.0.0.64",
  • verified MACs and IPs bound to node02 (Swarm worker using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest02 isaac.rodman]# MAC='7f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
[root@centos7swarmtest02 isaac.rodman]# MAC='7f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
Containers -
7d60ddb820dc:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.11",
                    "IPAddress": "10.0.0.130",
736be9a13208:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.129",
65364fde80c1:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.9",
                    "IPAddress": "10.0.0.128",
  • verified MACs and IPs bound to node03 (Swarm worker using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest03 isaac.rodman]# MAC='fb'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
Containers -
3d4051304e0f:    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.193",
c48496f97e40:    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.8",
                    "IPAddress": "10.0.0.192",
  • NOTE: if following this, you should now be able to browse to http://FQDN:5000 to vote on nodes which are running the gaiadocker/example-voting-app-vote:good container
  • NOTE: if following this, you should now be able to browse to http://FQDN:5001 to view results on nodes which are running the gaiadocker/example-voting-app-result:latest container

🎉 Cheers!

  • Side note 1: The follow was run after I installed the RPMs (above), and before I initialized the swarm on the master and workers.
    I am running the following /etc/docker/daemon.json and thinpool config. Please note that ipvlan requires experimental:true, and I am also running thinpool on CentOS 7.x, per the recommendations. My thinpool is on a VG called "docker" on the PV /dev/sdb (my second virtual drive on each node)
systemctl stop docker
 
rm -rf /var/lib/docker/*
 
vgremove -y docker
 
vgcreate docker /dev/sdb
 
echo y | lvcreate --wipesignatures y -n thinpool docker -l 95%VG
echo y | lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
 
lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
 
cat > /etc/lvm/profile/docker-thinpool.profile <<'EOF'
activation {
    thin_pool_autoextend_threshold=80
    thin_pool_autoextend_percent=20
}
EOF
 
lvchange --metadataprofile docker-thinpool docker/thinpool
 
# Verify the LV "thinpool" 
if lvs -o+seg_monitor docker | grep 'thinpool' | grep 'monitored'; then
  echo '* docker thinpool is monitored; SUCCESS'
else
  echo '* docker thinpool is not monitored; verify before starting Docker'
fi
 
cat > /etc/docker/daemon.json <<'EOF'
{
  "experimental": true,
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
  ]
}
EOF
 
systemctl start docker

eyz commented May 18, 2017

Hi all. First off: LGTM 👍 🎉

I had great success testing this PR #32981 with ipvlan on CentOS 7.x. The manpages still aren't building in the Makefile, but otherwise it is working great -

  • checked out of github.com/moby/moby 45c6f42 (was HEAD of master at the time of my checkout) into my go/src/github.com/moby, as go/src/github.com/moby/moby
  • removed all contrib/builder deb and rpm subfolders except contrib/builder/rpm/amd64/centos-7
isaac.rodman@ubuntu:/home/isaac/go/src/github.com/moby/moby# find contrib/builder/
contrib/builder/
contrib/builder/rpm
contrib/builder/rpm/amd64
contrib/builder/rpm/amd64/build.sh
contrib/builder/rpm/amd64/README.md
contrib/builder/rpm/amd64/generate.sh
contrib/builder/rpm/amd64/centos-7
contrib/builder/rpm/amd64/centos-7/Dockerfile
  • commented manpages in Makefile
#manpages: ## Generate man pages from go source and markdown
#       docker build ${DOCKER_BUILD_ARGS} -t docker-manpage-dev -f "man/$(DOCKERFILE)" ./man
#       docker run --rm \
#               -v $(PWD):/go/src/github.com/docker/docker/ \
#               docker-manpage-dev
  • attempted to make rpm (prior to further required modifications)
make rpm
  • waited for fail due to containerd tar not completing (below)
---> Making bundle: build-rpm (in bundles/17.06.0-dev/build-rpm)
Using test binary docker
+++ /etc/init.d/apparmor start
Starting AppArmor profiles:Warning from stdin (line 1): /sbin/apparmor_parser: cannot use or update cache, disable, or force-complain via stdin
.
INFO: Waiting for daemon to start...
+++ exec dockerd --debug --host unix:///go/src/github.com/docker/docker/bundles/17.06.0-dev/build-rpm/docker.sock --storage-driver btrfs --pidfile bundles/17.06.0-dev/build-rpm/docker.pid --userland-proxy=true
.
make: Nothing to be done for 'manpages'.
++ docker build -t dockercore/builder-rpm:centos-7 contrib/builder/rpm/amd64/centos-7/
Sending build context to Docker daemon   2.56kB
Step 1/10 : FROM centos:7
7: Pulling from library/centos
343b09361036: Pulling fs layer
343b09361036: Verifying Checksum
343b09361036: Download complete
343b09361036: Pull complete
Digest: sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077
Status: Downloaded newer image for centos:7
 ---> 8140d0c64310

[... elided ...]

Step 10/18 : WORKDIR /root/rpmbuild/SPECS
 ---> 941f5e39f6b8
Removing intermediate container 9065ac639458
Step 11/18 : RUN tar --exclude .git -r -C /usr/src -f /root/rpmbuild/SOURCES/docker-engine.tar docker-engine
 ---> Running in b917994182b2
 ---> e1657a479ebe
Removing intermediate container b917994182b2
Step 12/18 : RUN tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
 ---> Running in c2d757160a23
tar: containerd: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
The command '/bin/sh -c tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd' returned a non-zero code: 2
---> Making bundle: .integration-daemon-stop (in bundles/17.06.0-dev/build-rpm)
+++++ cat bundles/17.06.0-dev/build-rpm/docker.pid
++++ kill 3258
++++ /etc/init.d/apparmor stop
Clearing AppArmor profiles cache:.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
Makefile:144: recipe for target 'rpm' failed
make: *** [rpm] Error 1
  • commented these lines in bundles/17.06.0-dev/build-rpm/centos-7/Dockerfile.build
#RUN tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
#RUN tar --exclude .git -r -C /go/src/github.com/docker/libnetwork/cmd -f /root/rpmbuild/SOURCES/docker-engine.tar proxy
#RUN tar --exclude .git -r -C /go/src/github.com/opencontainers -f /root/rpmbuild/SOURCES/docker-engine.tar runc
#RUN tar --exclude .git -r -C /go/ -f /root/rpmbuild/SOURCES/docker-engine.tar tini
#RUN gzip /root/rpmbuild/SOURCES/docker-engine.tar
#RUN { cat /usr/src/docker-engine/contrib/builder/rpm/amd64/changelog; } >> docker-engine.spec && tail >&2 docker-engine.spec
#RUN rpmbuild -ba --define '_gitcommit 4874e05-unsupported' --define '_release 0.0.20170518.023124.git4874e05' --define '_version 17.06.0' --define '_origversion 17.06.0-dev' --define '_experimental 0' docker-engine.spec

^ removed extra whitespace on rpmbuild line after generated, to paste easier later

  • launched partial container (built without the lines we commented, above; we will run those manually after some modifications to the docker-engine.spec, below)
docker build -t docker-temp/build-rpm:centos-7 -f bundles/17.06.0-dev/build-rpm/centos-7/Dockerfile.build .
docker run -ti docker-temp/build-rpm:centos-7 bash
  • ensure you are in the /root/rpmbuild/SPECS folder inside the newly-built docker-temp/build-rpm:centos-7 container
cd /root/rpmbuild/SPECS
  • commented manpages lines in docker-engine.spec (first section)
# install manpages
#install -d %{buildroot}%{_mandir}/man1
#install -p -m 644 man/man1/*.1 $RPM_BUILD_ROOT/%{_mandir}/man1
#install -d %{buildroot}%{_mandir}/man5
#install -p -m 644 man/man5/*.5 $RPM_BUILD_ROOT/%{_mandir}/man5
#install -d %{buildroot}%{_mandir}/man8
#install -p -m 644 man/man8/*.8 $RPM_BUILD_ROOT/%{_mandir}/man8
  • commented docker CLI package in docker-engine.spec (second section; built from docker/cli master instead)
%files
%doc AUTHORS CHANGELOG.md CONTRIBUTING.md LICENSE MAINTAINERS NOTICE README.md
#/%{_bindir}/docker
  • commented manpages lines in docker-engine.spec (third section)
%doc
#/%{_mandir}/man1/*
#/%{_mandir}/man5/*
#/%{_mandir}/man8/*
  • manually ran commented lines from Dockerfile (except containerd and changelog append)
# tar --exclude .git -r -C /go/src/github.com/docker -f /root/rpmbuild/SOURCES/docker-engine.tar containerd
tar --exclude .git -r -C /go/src/github.com/docker/libnetwork/cmd -f /root/rpmbuild/SOURCES/docker-engine.tar proxy
tar --exclude .git -r -C /go/src/github.com/opencontainers -f /root/rpmbuild/SOURCES/docker-engine.tar runc
tar --exclude .git -r -C /go/ -f /root/rpmbuild/SOURCES/docker-engine.tar tini
gzip /root/rpmbuild/SOURCES/docker-engine.tar
# { cat /usr/src/docker-engine/contrib/builder/rpm/amd64/changelog; } >> docker-engine.spec && tail >&2 docker-engine.spec
rpmbuild -ba --define '_gitcommit 4874e05-unsupported' --define '_release 0.0.20170518.023124.git4874e05' --define '_version 17.06.0' --define '_origversion 17.06.0-dev' --define '_experimental 0' docker-engine.spec
  • observed that CentOS 7 RPMs were built
[root@63168ae7b4ef SPECS]# find /root/rpmbuild/RPMS/x86_64/ -type f
/root/rpmbuild/RPMS/x86_64/docker-engine-17.06.0-0.0.20170518.023124.git4874e05.el7.centos.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/docker-engine-debuginfo-17.06.0-0.0.20170518.023124.git4874e05.el7.centos.x86_64.rpm
  • copied built RPMs to nodes 01 thru 03
  • installed RPMs (nodes 01 thru 03, on CentOS 7.3)
yum -y install container-selinux
rpm -i docker-*.rpm
  • centos7swarmtest01 (Swarm master)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.64/26 -o parent=ens192 test_config_only # node 1
  • centos7swarmtest02 (Swarm worker)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.128/26 -o parent=ens192 test_config_only # node 2
  • centos7swarmtest03 (Swarm worker)
docker network create --config-only --subnet=10.0.0.0/24 --gateway=10.0.0.1 --ip-range=10.0.0.192/26 -o parent=ens192 test_config_only # node 3
  • centos7swarmtest01 (Swarm master)
docker network create -d ipvlan --scope=swarm --config-from test_config_only --attachable swarm_test
docker deploy -c /root/docker-compose-v3-test.yml voting
version: "3"

services:

  redis:
    image: redis:3.2-alpine
    ports:
      - "6379"
    networks:
      - votingnet
    deploy:
      placement:
        constraints: [node.role == manager]

  db:
    image: postgres:9.4
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - votingnet
    deploy:
      placement:
        constraints: [node.role == manager]

  voting-app:
    image: gaiadocker/example-voting-app-vote:good
    ports:
      - 5000:80
    networks:
      - votingnet
    depends_on:
      - redis
    deploy:
      mode: replicated
      replicas: 2
      labels: [APP=VOTING]
      placement:
        constraints: [node.role == worker]

  result-app:
    image: gaiadocker/example-voting-app-result:latest
    ports:
      - 5001:80
    networks:
      - votingnet
    depends_on:
      - db

  worker:
    image: gaiadocker/example-voting-app-worker:latest
    networks:
      votingnet:
        aliases:
          - workers
    depends_on:
      - db
      - redis
    # service deployment
    deploy:
      mode: replicated
      replicas: 2
      labels: [APP=VOTING]
      # service resource management
      resources:
        # Hard limit - Docker does not allow to allocate more
        limits:
          cpus: '0.25'
          memory: 512M
        # Soft limit - Docker makes best effort to return to it
        reservations:
          cpus: '0.25'
          memory: 256M
      # service restart policy
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      # service update configuration
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: continue
        monitor: 60s
        max_failure_ratio: 0.3
      # placement constraint - in this case on 'worker' nodes only
      placement:
        constraints: [node.role == worker]

networks:
    votingnet:
      external:
        name: swarm_test

volumes:
  db-data:
  • verified stack on centos7swarmtest01 (Swarm master)
[root@centos7swarmtest01 ~]# docker service ps --no-trunc voting
ID                          NAME                  IMAGE                                            NODE                DESIRED STATE       CURRENT STATE                     ERROR                                                             PORTS
51ebcxozsf8clwd5c3gj10l6c   voting_worker.1       gaiadocker/example-voting-app-worker:latest      centos7swarmtest02   Running             Running 29 minutes ago                                                                              
7ubvm4znlk2lls2sbw2orri7b   voting_result-app.1   gaiadocker/example-voting-app-result:latest      centos7swarmtest02   Running             Running 29 minutes ago                                                                              
znz47wst93kfic0lxlgdteoz6   voting_voting-app.1   gaiadocker/example-voting-app-vote:good          centos7swarmtest03   Running             Running 30 minutes ago                                                                              
r2f8r28z6nxe5bbk5t7wl2rbc   voting_db.1           postgres:9.4                                     centos7swarmtest01   Running             Running 29 minutes ago                                                                              
7xe7da400oycj3w0qs48d8xbk   voting_redis.1        redis:3.2-alpine                                 centos7swarmtest01   Running             Running 30 minutes ago                                                                              
zg97amwo70q800bgs11pe2h1p   voting_worker.2       gaiadocker/example-voting-app-worker:latest      centos7swarmtest03   Running             Running 29 minutes ago                                                                              
xeh62taucdw6uso3nrebogfp6   voting_voting-app.2   gaiadocker/example-voting-app-vote:good          centos7swarmtest02   Running             Running 29 minutes ago                   
  • verified MACs and IPs bound to centos7swarmtest01 (Swarm master using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest01 ~]# MAC='4f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
Containers -
4e50f691d32c:    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.65",
3bbc02a6a682:    link/ether 00:50:56:bc:74:4f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.6",
                    "IPAddress": "10.0.0.64",
  • verified MACs and IPs bound to node02 (Swarm worker using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest02 isaac.rodman]# MAC='7f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
[root@centos7swarmtest02 isaac.rodman]# MAC='7f'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
Containers -
7d60ddb820dc:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.11",
                    "IPAddress": "10.0.0.130",
736be9a13208:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.129",
65364fde80c1:    link/ether 00:50:56:bc:7b:7f brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.9",
                    "IPAddress": "10.0.0.128",
  • verified MACs and IPs bound to node03 (Swarm worker using ipvlan global network), as well as that the IPs successfully were picked from the host's IPRange, as configured
[root@centos7swarmtest03 isaac.rodman]# MAC='fb'; echo 'Host - '; ip addr show ens192 | grep "$MAC"; echo 'Containers -'; for CID in $(docker ps -a -q); do echo -n "$CID:"; docker exec -ti $CID ip addr | grep "$MAC"; docker inspect $CID | grep 'IPAddress'; done
Host - 
    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
Containers -
3d4051304e0f:    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.193",
c48496f97e40:    link/ether 00:50:56:bc:43:fb brd ff:ff:ff:ff:ff:ff
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.255.0.8",
                    "IPAddress": "10.0.0.192",
  • NOTE: if following this, you should now be able to browse to http://FQDN:5000 to vote on nodes which are running the gaiadocker/example-voting-app-vote:good container
  • NOTE: if following this, you should now be able to browse to http://FQDN:5001 to view results on nodes which are running the gaiadocker/example-voting-app-result:latest container

🎉 Cheers!

  • Side note 1: The follow was run after I installed the RPMs (above), and before I initialized the swarm on the master and workers.
    I am running the following /etc/docker/daemon.json and thinpool config. Please note that ipvlan requires experimental:true, and I am also running thinpool on CentOS 7.x, per the recommendations. My thinpool is on a VG called "docker" on the PV /dev/sdb (my second virtual drive on each node)
systemctl stop docker
 
rm -rf /var/lib/docker/*
 
vgremove -y docker
 
vgcreate docker /dev/sdb
 
echo y | lvcreate --wipesignatures y -n thinpool docker -l 95%VG
echo y | lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
 
lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
 
cat > /etc/lvm/profile/docker-thinpool.profile <<'EOF'
activation {
    thin_pool_autoextend_threshold=80
    thin_pool_autoextend_percent=20
}
EOF
 
lvchange --metadataprofile docker-thinpool docker/thinpool
 
# Verify the LV "thinpool" 
if lvs -o+seg_monitor docker | grep 'thinpool' | grep 'monitored'; then
  echo '* docker thinpool is monitored; SUCCESS'
else
  echo '* docker thinpool is not monitored; verify before starting Docker'
fi
 
cat > /etc/docker/daemon.json <<'EOF'
{
  "experimental": true,
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
  ]
}
EOF
 
systemctl start docker
@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo May 18, 2017

Contributor

@eyz thanks a lot for validating this PR. The logs you posted will be very helpful for others to follow.

Contributor

mavenugo commented May 18, 2017

@eyz thanks a lot for validating this PR. The logs you posted will be very helpful for others to follow.

@albers

This comment has been minimized.

Show comment
Hide comment
@albers

albers May 18, 2017

Member

Also /cc @albers @sdurrheimer for completion script changes involved

@thaJeztah I'm still waiting for the completions to be moved to docker/cli.

Member

albers commented May 18, 2017

Also /cc @albers @sdurrheimer for completion script changes involved

@thaJeztah I'm still waiting for the completions to be moved to docker/cli.

@eyz

This comment has been minimized.

Show comment
Hide comment
@eyz

eyz May 27, 2017

I believe I found a bug with the combination of --config-only and --ipam-driver (see below):

  • when no --config-only flag is specified, --ipam-driver is used properly and I get a subnet and gateway from the IPAM driver
  • when setting --config-only, the IPAM.Driver appears to not be set even though it is specified in my example with --ipam-driver
  • I cannot specify --ipam-driver when creating the global-scoped network (as expected) and get the expected error

See: ipam-driver cannot be used with config-only network #33415

eyz commented May 27, 2017

I believe I found a bug with the combination of --config-only and --ipam-driver (see below):

  • when no --config-only flag is specified, --ipam-driver is used properly and I get a subnet and gateway from the IPAM driver
  • when setting --config-only, the IPAM.Driver appears to not be set even though it is specified in my example with --ipam-driver
  • I cannot specify --ipam-driver when creating the global-scoped network (as expected) and get the expected error

See: ipam-driver cannot be used with config-only network #33415

@smakam

This comment has been minimized.

Show comment
Hide comment
@smakam

smakam Jun 13, 2017

hi
I have few questions regarding this PR. pls correct me if this is not the correct place to ask question on the PR.
For networks that are capable of providing connectivity across hosts like macvlan and ipvlan, we can create services and provide connectivity across services using this PR. For node local network like bridge or host, how do we provide connectivity across services? Will Swarm make sure to schedule the service in same node? I did not see it happening.
Is it correct that we dont need to specify "config-only" and "config-from" for bridge, user defined bridge and host network if we keep default option for the networks?
Is it correct that node local network with swarm will support service of only dns rr with publish mode host?

Thanks
Sreenivas

smakam commented Jun 13, 2017

hi
I have few questions regarding this PR. pls correct me if this is not the correct place to ask question on the PR.
For networks that are capable of providing connectivity across hosts like macvlan and ipvlan, we can create services and provide connectivity across services using this PR. For node local network like bridge or host, how do we provide connectivity across services? Will Swarm make sure to schedule the service in same node? I did not see it happening.
Is it correct that we dont need to specify "config-only" and "config-from" for bridge, user defined bridge and host network if we keep default option for the networks?
Is it correct that node local network with swarm will support service of only dns rr with publish mode host?

Thanks
Sreenivas

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jun 13, 2017

Member

@smakam when using bridge, swarm does not do any magic, just attaches the tasks to the network, but no overlay / cross-node communication.

Member

thaJeztah commented Jun 13, 2017

@smakam when using bridge, swarm does not do any magic, just attaches the tasks to the network, but no overlay / cross-node communication.

@smakam

This comment has been minimized.

Show comment
Hide comment
@smakam

smakam Jun 14, 2017

@thaJeztah Thanks for the response. Yes, I understand that we cannot do cross-node communication with bridge network. What I see is that Swarm is not aware of node local network or node global network(dont know if its right term) and it can schedule services across nodes and connectivity will not work. Do we take care of this by specifying constraints so that services in node local network get scheduled in same node or will Swarm be able to do it automatically if its node local network?

smakam commented Jun 14, 2017

@thaJeztah Thanks for the response. Yes, I understand that we cannot do cross-node communication with bridge network. What I see is that Swarm is not aware of node local network or node global network(dont know if its right term) and it can schedule services across nodes and connectivity will not work. Do we take care of this by specifying constraints so that services in node local network get scheduled in same node or will Swarm be able to do it automatically if its node local network?

@lbguilherme

This comment has been minimized.

Show comment
Hide comment
@lbguilherme

lbguilherme Jun 18, 2017

Not quite the right place to ask, but it's related to this feature: Is there a way for containers of a service to be present on both the host network and an overlay network? My use case is that I have a HTTP server doing reverse-proxing and tls termination of several services. The problem is that if I put my service in the overlay and publish the port, then there is no way to know the IP of the remote client #25526. But if I put this service in the host network, then I can address all other services to relay the traffic.

Container cannot be disconnected from host network or connected to host network

Look's like a dead end.

lbguilherme commented Jun 18, 2017

Not quite the right place to ask, but it's related to this feature: Is there a way for containers of a service to be present on both the host network and an overlay network? My use case is that I have a HTTP server doing reverse-proxing and tls termination of several services. The problem is that if I put my service in the overlay and publish the port, then there is no way to know the IP of the remote client #25526. But if I put this service in the host network, then I can address all other services to relay the traffic.

Container cannot be disconnected from host network or connected to host network

Look's like a dead end.

@groyee

This comment has been minimized.

Show comment
Hide comment
@groyee

groyee Jun 30, 2017

I completely got lost here.

Is it possible to create a docker service with a (--net=host) as it used to be?

I don't want to create any overlay networks.

groyee commented Jun 30, 2017

I completely got lost here.

Is it possible to create a docker service with a (--net=host) as it used to be?

I don't want to create any overlay networks.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 1, 2017

Member

Keep in mind that the GitHub issue tracker is not intended as a general support forum,
but for reporting bugs and feature requests. For other type of questions, consider using one of;

I'm locking the discussion on this PR. If you think there's a bug in this feature, please open a bug report

Member

thaJeztah commented Jul 1, 2017

Keep in mind that the GitHub issue tracker is not intended as a general support forum,
but for reporting bugs and feature requests. For other type of questions, consider using one of;

I'm locking the discussion on this PR. If you think there's a bug in this feature, please open a bug report

@moby moby locked and limited conversation to collaborators Jul 1, 2017

@aboch aboch deleted the aboch:nlo2 branch Nov 8, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.