New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macvlan network support in Rancher cattle #8686

Closed
niusmallnan opened this Issue May 6, 2017 · 7 comments

Comments

Projects
None yet
10 participants
@niusmallnan
Member

niusmallnan commented May 6, 2017

I have updated network-manager, it can be able to support other CNI drivers. So macvlan network support can be achieved. #8535

By the customer's communication, there are two main scenarios using macvlan. One is "Inner Gateway Mode", another is "External Gateway Mode", if names are not good please forgive me.😄

Inner Gateway Mode
image
I call it "Inner Gateway Mode", because users do not need to configure the gateway for macvlan subnet.
Notice:

  1. containers subnet is 192.168.22.0/24
  2. rancher-md-gw is a macvlan device, its parent link is eth0(eth0.x)
  3. we use macvlan bridge mode, so an IP address is not required on host interface eth0.
  4. for the containers to access the external network, we need to add SNAT rules.

External Gateway Mode
image
I call it "External Gateway Mode", because there will be a physical router in user's network.
Notice:

  1. containers subnet is 192.168.22.0/24
  2. rancher-md-gw is a macvlan device, its parent link is eth0(eth0.x). But I set a reserved IP address on it(like 192.168.22.254).
  3. each container has a special route record, dst 169.254.169.250 via 192.168.22.254.

Basically, most users are looking forward to the first mode, because there is no need to transform their network too much. We bring about the first mode, if some people need second mode, the transformation will be very easy.

For catalog item:
Repo: https://github.com/niusmallnan/flatnet-catalog.git
Branch: test
Item: macvlan

For rancher-cni-macvlan:
Repo: https://github.com/niusmallnan/rancher-cni-macvlan

@danfromtitan

This comment has been minimized.

Show comment
Hide comment
@danfromtitan

danfromtitan Jun 2, 2017

I want to use macvlan driver to attach containers on Internet routable IP addresses and at the same time avoid Internet traffic to reach the host itself. For that purpose I resort on L2 VLAN separation, thus having a dedicated public vlan interface on the host (no IP address on the host public interface though).

Couple of questions:

  • how do I go about enslaving a selected vlan interface for the macvlan driver (i.e. eth0.xyz)
  • instead of rancher-md-gw, can I use a private ipsec overlay network and have containers with two interfaces, one in the public network and the other in the private overlay network (for dns and metadata access).

danfromtitan commented Jun 2, 2017

I want to use macvlan driver to attach containers on Internet routable IP addresses and at the same time avoid Internet traffic to reach the host itself. For that purpose I resort on L2 VLAN separation, thus having a dedicated public vlan interface on the host (no IP address on the host public interface though).

Couple of questions:

  • how do I go about enslaving a selected vlan interface for the macvlan driver (i.e. eth0.xyz)
  • instead of rancher-md-gw, can I use a private ipsec overlay network and have containers with two interfaces, one in the public network and the other in the private overlay network (for dns and metadata access).
@jnovack

This comment has been minimized.

Show comment
Hide comment
@jnovack

jnovack Jul 1, 2017

I'm in the "second mode"/"external gateway" mode.

I have containers, that I want to be on the rancher network (192.168.22.0/24, from the example) because I want them to have a "known" IP address regardless of which host they are on. Assuming rancher-host is 192.168.22.101.

Let's assume options of mitigation (load-balancing, service-discovery, SNI forwarding) have been exhausted, I'm not trying to diagnose WHY one would use this in one's environment, just grant the premise that one does. Static NATs, mac-address pinning, tight ACLs; whatever the reason.

In this scenario, docker-compose version 2 permits mac_address. So we set a mac_address and put it on the macvlan/rancher-host. It gets a parallel address to rancher-host (e.g. 192.168.22.240) via the 192.168.22.0/24 DHCP server. If and when I evacuate this service, it's movement to another rancher-host (e.g. 192.168.22.102) would yield it's same ip address 192.168.22.240).

Currently, this is very easily do-able in a non-rancher environment. but not scalable/managable in rancher.

docker network create -d macvlan --subnet=192.168.22.0/24 --gateway=192.168.22.1 --ip-range=192.168.22.240/29 --ipv6 --subnet=2001:abcd:1234::/64 -o parent=eth0 mynet

jnovack commented Jul 1, 2017

I'm in the "second mode"/"external gateway" mode.

I have containers, that I want to be on the rancher network (192.168.22.0/24, from the example) because I want them to have a "known" IP address regardless of which host they are on. Assuming rancher-host is 192.168.22.101.

Let's assume options of mitigation (load-balancing, service-discovery, SNI forwarding) have been exhausted, I'm not trying to diagnose WHY one would use this in one's environment, just grant the premise that one does. Static NATs, mac-address pinning, tight ACLs; whatever the reason.

In this scenario, docker-compose version 2 permits mac_address. So we set a mac_address and put it on the macvlan/rancher-host. It gets a parallel address to rancher-host (e.g. 192.168.22.240) via the 192.168.22.0/24 DHCP server. If and when I evacuate this service, it's movement to another rancher-host (e.g. 192.168.22.102) would yield it's same ip address 192.168.22.240).

Currently, this is very easily do-able in a non-rancher environment. but not scalable/managable in rancher.

docker network create -d macvlan --subnet=192.168.22.0/24 --gateway=192.168.22.1 --ip-range=192.168.22.240/29 --ipv6 --subnet=2001:abcd:1234::/64 -o parent=eth0 mynet

@jnovack

This comment has been minimized.

Show comment
Hide comment
@jnovack

jnovack Jul 1, 2017

each container has a special route record, dst 169.254.169.250 via 192.168.22.254.

Can you link the containers purposefully within Rancher so you don't have to waste another IP per Host for a docker0 gateway?

web:
  links:
    - rancher-metadata:{{r-network-services-metadata-container}}

Or even the use of sidekick containers (yes, 1 per) may be preferable over using another IPv4 address on the main subnet (which may not be yours to hand out).

jnovack commented Jul 1, 2017

each container has a special route record, dst 169.254.169.250 via 192.168.22.254.

Can you link the containers purposefully within Rancher so you don't have to waste another IP per Host for a docker0 gateway?

web:
  links:
    - rancher-metadata:{{r-network-services-metadata-container}}

Or even the use of sidekick containers (yes, 1 per) may be preferable over using another IPv4 address on the main subnet (which may not be yours to hand out).

@kvaps

This comment has been minimized.

Show comment
Hide comment
@kvaps

kvaps Jul 11, 2017

Hi, for me I solved this issue by pipework script. Just create global service:

version: '2'
services:
  pipework:
    privileged: true
    image: dreamcat4/pipework
    environment:
      host_routes: 'true'
      route_add_delay: '1'
      run_mode: batch,daemon
    network_mode: host
    volumes:
    - /var/run/docker.sock:/docker.sock
    pid: host
    labels:
      io.rancher.scheduler.global: 'true'

After it, you can run other containers and specify any custom network settings for it by simple adding environment variable to it, example:

version: '2'
services:
  test:
    image: ubuntu:14.04.3
    environment:
      pipework_cmd: br0 @CONTAINER_NAME@ 10.36.60.9/16@10.36.0.1

After starting, the new interface will be added into your container, which connected to bridge br0, and the ip 10.36.60.9/16 will be assigned to it, default gateway will be changed to 10.36.0.1.

More examples here:
https://github.com/dreamcat4/docker-images/blob/master/pipework/3.%20Examples.md

kvaps commented Jul 11, 2017

Hi, for me I solved this issue by pipework script. Just create global service:

version: '2'
services:
  pipework:
    privileged: true
    image: dreamcat4/pipework
    environment:
      host_routes: 'true'
      route_add_delay: '1'
      run_mode: batch,daemon
    network_mode: host
    volumes:
    - /var/run/docker.sock:/docker.sock
    pid: host
    labels:
      io.rancher.scheduler.global: 'true'

After it, you can run other containers and specify any custom network settings for it by simple adding environment variable to it, example:

version: '2'
services:
  test:
    image: ubuntu:14.04.3
    environment:
      pipework_cmd: br0 @CONTAINER_NAME@ 10.36.60.9/16@10.36.0.1

After starting, the new interface will be added into your container, which connected to bridge br0, and the ip 10.36.60.9/16 will be assigned to it, default gateway will be changed to 10.36.0.1.

More examples here:
https://github.com/dreamcat4/docker-images/blob/master/pipework/3.%20Examples.md

@niusmallnan niusmallnan self-assigned this Aug 8, 2017

@pwFoo

This comment has been minimized.

Show comment
Hide comment
@pwFoo

pwFoo Sep 8, 2017

@niusmallnan
So your second approach would add routable subnet(s) to rancher like added to libnetwork in docker version 1.10+?
image

libnetwork with docker 1.10+

# create a new bridge network with your subnet and gateway for your ip block
$ docker network create --subnet 203.0.113.0/24 --gateway 203.0.113.254 iptastic

# run a nginx container with a specific ip in that block
$ docker run --rm -it --net iptastic --ip 203.0.113.2 nginx

Would be great if rancher server could support the docker network features native?

pwFoo commented Sep 8, 2017

@niusmallnan
So your second approach would add routable subnet(s) to rancher like added to libnetwork in docker version 1.10+?
image

libnetwork with docker 1.10+

# create a new bridge network with your subnet and gateway for your ip block
$ docker network create --subnet 203.0.113.0/24 --gateway 203.0.113.254 iptastic

# run a nginx container with a specific ip in that block
$ docker run --rm -it --net iptastic --ip 203.0.113.2 nginx

Would be great if rancher server could support the docker network features native?

@stepman0

This comment has been minimized.

Show comment
Hide comment
@stepman0

stepman0 Jul 16, 2018

Stupid question: How do I use this plugin? I installed it from catalog, but found no description on how to use this cni-plugin for any of my containers / stacks.

stepman0 commented Jul 16, 2018

Stupid question: How do I use this plugin? I installed it from catalog, but found no description on how to use this cni-plugin for any of my containers / stacks.

@loganhz

This comment has been minimized.

Show comment
Hide comment
@loganhz

loganhz Oct 12, 2018

Member

Thanks for your report!

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.

If you think we should keep this issue open, please let me know.

Member

loganhz commented Oct 12, 2018

Thanks for your report!

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.

If you think we should keep this issue open, please let me know.

@loganhz loganhz closed this Oct 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment