Revamp mgmt networking

[plajjan]

We are currently doing a rather fugly hack for the mgmt interface of each container. Ultimately it makes the mgmt interface available on the docker IP address, which I think is a good thing, but how we get there isn't very beautiful.

The virtual router has a mgmt interface which we always configure the address 10.0.0.15 on. This interface is connected to a qemu "host interface" which uses host forwarding to forward some ports (like SSH) from a listen socket that qemu opened in the docker container, to the virtual machine. Normally when qemu does this forwarding it doesn't rewrite the source address, it's just doing destination NAT, so the virtual machine still has to have a return route, normally a default route, pointing back to qemu (10.0.0.1). Since that might interfere with configuration the user wants to apply to his virtual router, we can't do that. So we use socat in the docker container to proxy all connections, effectively rewriting the source address so we don't need a default route in the virtual router.

Anyway, what I want to do is to put the IP address of the container directly on the virtual router. Or rather, I want it to think it has the address but at the same time keep the address configured in the docker container, i.e. both container and the virtual machine would have the same address. We should then have some form of pseudo transparent bridge which can steer traffic either to the docker container or to the virtual router based on the destination port. I want some ports to go the the docker container so we can reach qemu ports there, like port 4000 (qemu monitor), 5000 (serial console) and 10000+ since those are for our own overlay network. The rest of the ports, like 22, udp/161, 830 etc go to the virtual router so it can respond to SSH etc.

Coincidentally I saw that rancherVM is doing something very similar, in the launch script of the container they capture the MAC & IP, then deconfigure it from the docker interface, setup a bridge and a DHCP server that serves this address, then spin up the virtual machine and let DHCP hand out the address to the VM. The result is that the VM is directly connected to what used to be the containers interface, using the same IP etc. Very cool. I want the same but in addition doing this pseudo-transparent bridging with traffic steering so that we can control what should serve a given port.

[plajjan]

I think we can implement that magic bridge using Snabb.

[davidban77]

Hey @plajjan thanks for this great tool BTW!

I find myself on this issue trying to understand a little bit better the work behind the mgmt interface and docker container. Currently I would like to enable logging on the virtual router and have that traffic go to the external world.

I was browsing around the issues and I couldn't find an approach to this scenario.

Maybe the workaround could be to setup the log collection as an extra docker container and use vr-xcon tcp2tap to connect the 2 of them?

Do you have recommendation on this scenario?