Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Support for private networks #120

Closed
fgrehm opened this Issue · 31 comments
@fgrehm
Owner
@stormbrew

Not that this is helpful to getting it done, but this is the only thing keeping me from using vagrant-lxc right now. Really excited for this to get in there.

@fgrehm
Owner

@stormbrew no worries! Networking "stuff" will probably be my next focus once 0.5.0 is out ;)

@fgrehm
Owner

@stormbrew 0.5.0 is out :)

So, regarding this feature, I'm thinking about how we should implement it and I'd love a second opinion on something related to this. I'm not sure you know but Vagrant configures an additional network interface for VBox hosts besides the one with the IP you've specified.

Do you think we should have that same behavior here or should it just mean that we'll set lxc.network.ipv4 when starting the container? Another idea would be to use @paneq's approach of writing to /etc/network/interfaces.

I'm no networking expert here but I'm up for implementing the support for it. So I'd love some feedback about this :)

/cc @rcarmo

@stormbrew

The best thing would be for it to be as much like existing (VBox) vagrant setups as possible. So eth0 inside the container is either NAT or bridge (or even, if my understanding of what's possible with lxc is correct, the exact same interface as outside) and then each config.network creates another interface as specified (with private being bridged such that other containers on the same subnet can communicate with it).

@fgrehm
Owner

@stormbrew tks for the input.

regarding the default eth0, right now the "official" boxes link eth0 to lxcbr0 as it is the default bridge that comes with Ubuntu's lxc package and (if I'm not mistaken) is NATted. That is nice for Ubuntu users but it kinda sucks as other distros don't come with that by default. I'm planning to drop that requirement and / or add support for setting up and start using a vagrantlxcbr automagically once I'm able to deal with mitchellh/vagrant#2005. this is something I want to see happening before our 1.0

regarding the private interfaces configuration and bridge setup, what you are saying is that we could set up another bridge per subnet and manually configure the interfaces? would you be able to come up with a bash script that does that set up on the host and on the container so that we can find our way of making it happen from the plugin? If you need a sandbox to try things out you might want to use one of the boxes available at https://github.com/fgrehm/vagrant-lxc-vbox-hosts ;)

@rcarmo
@oker1

I think the private network should work this way (pretty similar to the vbox provider with the host only interfaces):

  1. list all lxcbr(\d+) interfaces where $1 > 0
    • if any of them has the same subnet as the private ip choose it
    • otherwise create a bridge with brctl addbr lxcbrX where X is the next free number, then ifconfig lxcbrX x.y.z.1 255.255.255.0 up
  2. add interface to lxc template like this:
      lxc.customize "network.type", "veth"
      lxc.customize "network.flags", "up"
      lxc.customize "network.link", "lxcbrX"
      lxc.customize "network.ipv4", "ip/24"

Doing this manually works correctly, the vms can ping each other on their eth1s. They can ping the host and the host can ping them too.

@fgrehm
Owner

@oker1 tks the the input, I like the idea :) I'll be looking into mitchellh/vagrant#2005 any time "soon", but since 1.3.0 will probably going to take a while to be released I think we'll need to find our way to make it happen without host capabilities if we want to have this working with 1.1+.
I'll have a look into this when I have a chance and I'd be really happy if someone is able to send a PR ;)

@oker1

Cool, one more addition: when a vm with private network was shut down, the provider should check the bridge belonging to the network and if it has no interfaces attached to it, the bridge should be downed, so we dont leave unused bridges on the host. It has no interfaces attached if /sys/class/net/${LXC_BRIDGE}/brif/ directory is empty.

@jellonek
@oker1

I think it's tidier not to leave unused interfaces around. Bringing it up does not take significant time. But you are right, it's not crucial.

@stormbrew

I wonder if taking it down may interfere with services bound to the bridge's ip (explicitly, as opposed to INADDR_ANY) between restarts of a container. That would be undesirable.

@fgrehm
Owner

tks guys! as I said my networking skills are pretty limited and right now I don't have a need for this. Whatever you guys think is best I'm up for implementing it. If someone is able to send a PR I'll be more that happy to merge it in ;)

@pleddy

http://sysadminandnetworking.blogspot.com/2013/09/set-ip-on-vagrant-lxc-vm.html

Please, feel free to ask me networking questions, I know a bit. And, can research quick if I don't have direct experience.

@fgrehm
Owner

@pleddy cool, I need to fix a couple of bugs related to 0.6.0 and I'll look into networking related stuff after that :)

@fgrehm
Owner

I've asked this for some of you on twitter but is anyone familiar with pipework? Does anyone think we could use it to get this implemented?

@kikitux
@fgrehm fgrehm modified the milestone: v0.9.0, v1.0.0
@fgrehm
Owner

I gave a shot at using pipework and it works! Here's what it would take to create a private network for a container for future reference:

pipework br1 <container-id> 192.168.1.1/24
ip addr add 192.168.1.254/24 dev br1
@stucki

Looks good! I had a look at Vagrant + VirtualBox, and it seems like it does a similar thing on the host. It always creates a /24 network for the given private IP.

Since pipework was originally written for use with Docker, I suggest to use only the necessary commands and include them directly in vagrant-lxc...

Additionally, during shutdown, I suggest to run ip link delete br1 in order to clean up again...

@stucki

Oh, and I suggest to use a meaningful name for the bridge interface, e.g. "vagrant" rather than "br1".

@fgrehm
Owner

@stucki thanks for the feedback!

As per getting rid of Docker's related code from pipework, I'm not sure I'll have the time to get that into 1.0. My plan is to actually not even ship with pipework and ask users to install it by hand on 0.9.0 (which will probably be the 1.0.0 feature without dropping support for vagrant < 1.5) and look into polishing things up afterwards.
The reason behind it is that it is a feature that I'm not going to use on a daily basis and would love some feedback from the community whether it works before going that direction ;-)

In regard to ip link delete br1, I'm not 100% sure we can do that, we might have other vagrant-lxc containers attached to the same bridge on a different Vagrantfile (or even "vanilla" containers) that would stop working. IIRC, (at least) Ubuntu will not keep those bridges around on the host and pipework takes care of "garbage collecting" container's veths. I think that's good enough for now.

And for the bridge name, I'm actually planning to make that configurable but defaulting it to brX as we might end with with many bridges for different networks as pointed out by @oker1 on #120 (comment)

As I said many times, I'm not a networking guru, so please correct me if I'm wrong about anything, this is the best time to :punch: me before the feature ends up on master :smiley:

@stucki

You can easily find out if the bridge is still used by checking if it has any interfaces attached to it using brctl show.

Bridge name: As it looks to me, VirtualBox is creating vboxnet<number> interfaces and increments the number for each distinct network. That seems like a good way of dealing with it and can safe another config option...

Regarding the integration of pipework, I think my Ruby skills are not good enough to code that myself. However, I assume that if you get pipework running through an external command, then I would be able to do the rest, so that we don't depend on having pipework installed...

While we're at it, make sure to add a note that the whole pipework stuff depends on the "brctl" utility. On Ubuntu it is the bridge-utils package which includes that.

Greetings, and thanks for your amazing work!

@fgrehm
Owner

You can easily find out if the bridge is still used by checking if it has any interfaces attached to it using brctl show.

Awesome! Thanks for sharing!

While we're at it, make sure to add a note that the whole pipework stuff depends on the "brctl" utility. On Ubuntu it is the bridge-utils package which includes that.

Sure!

Greetings, and thanks for your amazing work!

Thank you for helping out :-)

@fgrehm fgrehm modified the milestone: v1.0.0, v0.9.0
@cbanbury

bump!

Apologies for nagging, is this still being worked on? I'm currently implementing this manually but wondering if it's worth waiting for this to be included before doing the same to all my Vagrantfiles.

Thanks :smiley_cat:

@fgrehm
Owner

This is not being worked on right now but will be a feature of the first 1.0.0 beta when that comes out ;)

@bbinet

+1, this would be a really useful addition.

@bpaz

@cbanbury can you show how you are implementing this manually?

@cbanbury

@bpaz sorry for not replying sooner, I don't know if this is still any use to you but here's what I'm doing:

lxc.customize "network.type", "veth"
lxc.customize "network.link", "lxcbr1"
lxc.customize "network.ipv4", "10.0.3.x/24"
This was referenced
@fgrehm fgrehm removed this from the v1.0.0 milestone
@fgrehm fgrehm modified the milestone: v1.1.0
@fgrehm
Owner

Experimental support for this has landed on git and will be part of 1.1.0 :fireworks:

More info on #298 (comment)

@fgrehm fgrehm closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.