New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to select which default gateway to use when multiple macvlan nets attached #35221

Open
killcity opened this Issue Oct 16, 2017 · 2 comments

Comments

Projects
None yet
4 participants
@killcity

killcity commented Oct 16, 2017

Description

When attaching multiple macvlan nets on docker service create (one external, one internal), the internal is always chosen. There is no way to choose the external for the default route.

Steps to reproduce the issue:

  1. create two macvlan swarm nets, separate nets
  2. create service using and attach both nets
  3. check route table and you will see its selects a route

Describe the results you received:

default behavior:

ifconfig -a

eth0      Link encap:Ethernet  HWaddr 02:42:0A:F1:01:82
          inet addr:10.241.1.130  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:52313 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50601 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:32512621 (31.0 MiB)  TX bytes:5352903 (5.1 MiB)

eth1      Link encap:Ethernet  HWaddr 02:42:43:E2:D2:21
          inet addr:192.0.0.25  Bcast:0.0.0.0  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

netstat -rn

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.241.0.1      0.0.0.0         UG        0 0          0 eth0
10.241.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.0.0.25    0.0.0.0         255.255.254.0   U         0 0          0 eth1

Describe the results you expected:
I expected the above results.

The problem is that I want the following route table instead:

netstat -rn

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0      67.226.210.1    255.255.254.0   U         0 0          0 eth1
10.241.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
67.226.210.0    0.0.0.0         255.255.254.0   U         0 0          0 eth1

Additional information you deem important (e.g. issue happens only occasionally):
Regardless of order, the RFC1918 ip is always chosen. This is an issue when you are trying to run a container with a public ip and an internal ip.

HACK:
The only workaround for this, is to ditch the use of Swarm, since swarm doesn't support --privileged. I need --privileged in order to run a script when the containers starts (via docker run). The script will juggle the routes and we will end up with the desired state. So unless we can select a specific gateway (and add addtional routes) or get --privileged added to docker service, it's not easy running a public/private container with swarm + macvlan.

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 20:00:25 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 10
 Running: 3
 Paused: 0
 Stopped: 7
Images: 16
Server Version: 17.06.2-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: false
 Native Overlay Diff: true
Logging Driver: gelf
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: r6exfwgg7p4837ytv7746sntz
 Is Manager: true
 ClusterID: ejo78ur1erlw6azupui3wy175
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.242.0.25
 Manager Addresses:
  10.242.0.23:2377
  10.242.0.24:2377
  10.242.0.25:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.12.10-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 11.73GiB
Name: dockerhost-test-nb5h9zw3
ID: X75E:3H7J:23MA:L7C2:KXIY:IS7I:KI4H:2PK7:GVS2:AGRM:2A2P:L56F
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
This is running under VMware, but results are the same on bare metal.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Oct 31, 2017

Slightly related as well;

  • Unable to choose outbound (external) IP for containers #30053
  • Allow use of network name for network interface #23742
  • Control interface name in docker multiple networking #25181
@jsenecal

This comment has been minimized.

jsenecal commented May 7, 2018

Facing the same issue - for a VPN container with a leg inside private network and outside at the same time...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment