Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with virtualbox provisioner private_network mode in cloud ubuntu 14.04 box #3860

Closed
zerkms opened this issue May 20, 2014 · 52 comments
Closed

Comments

@zerkms
Copy link

@zerkms zerkms commented May 20, 2014

There are some issues related to cloud-init-nonet caused by virtualbox provisioner:

If you create a vm from ubuntu/trusty64 box.

Then after it's up and running add

config.vm.network "private_network", type: :dhcp

private network, then after 2 reboots using vagrant reload the cloud-init-nonet process will spend 120 seconds waiting for something, followed by another 60 seconds from some other startup script. Eventually it will start but it will take at least 3 minutes.

Something relevant I've found:

https://bugs.launchpad.net/cloud-init/+bug/1315501
https://ask.openstack.org/en/question/28297/cloud-init-nonet-waiting-and-fails/

@mitchellh
Copy link
Member

@mitchellh mitchellh commented May 21, 2014

I think this is just a bug with ubuntu, not with Vagrant. Please correct me if I'm wrong and advise on what approach Vagrant should take.

@mitchellh mitchellh closed this May 21, 2014
@zerkms
Copy link
Author

@zerkms zerkms commented May 21, 2014

Well, I turned off the vagrant network configuration for host-only adapter using auto_config: false and just put

auto eth1
iface eth1 inet dhcp

to the /etc/network/interfaces.d/eth1.cfg.

Which solved the issue.

I assume that's what should be done by vagrant not manually.

@mitchellh
Copy link
Member

@mitchellh mitchellh commented May 21, 2014

That is what Vagrant does. It sounds like with this bug though on restart Ubuntu/cloud-init is actually changing that to a static definitoin.

@zerkms
Copy link
Author

@zerkms zerkms commented May 21, 2014

Well, the thing is if it's added manually in a separate file - it works. If added by vagrant into a common /etc/network/interfaces - it doesn't (there is a difference though - I didn't drop the route as vagrant does, may be that is the issue).

I'm not confident enough to tell if it's a vagrant or that particular image or cloud-init or whatever else bug :-)

@mitchellh
Copy link
Member

@mitchellh mitchellh commented May 21, 2014

Ah I see what you're saying. Okay, let me take a closer look.

@zerkms
Copy link
Author

@zerkms zerkms commented May 28, 2014

Any news on investigation?

Just experienced exactly the same on ubuntu 12.04 and bridged adapter...

@mitchellh
Copy link
Member

@mitchellh mitchellh commented May 28, 2014

Not yet, changing the networking stuff is quite scary because it isn't well tested (hard to test this), so work on this won't go until 1.7.

@Houndie
Copy link

@Houndie Houndie commented Jun 12, 2014

I can confirm, I also experience the same issue.

EDIT: Okay so it appears the issue is the line

post-up route del default dev $IFACE

Commenting this line out fixes the problem. I'll admit, I'm not really network savvy, so I'm not sure what this line does, or why it's important, or why we need it...but there's probably a good reason, and commenting it out is probably not the best solution.

Either way, until the problem is either fixed, or someone can tell me why the band-aid of deleting that line is a bad-idea, my temporary work-around is to have my provisioner delete the line.


EDIT2: Never mind, vagrant is a jerk and keeps adding it back in because auto config is on (understandable)...the best band-aid would probably be to add it to the init process then, but now I'm getting to really hacky territory--it would be nice if a real solution was found.


EDIT3: Okay, I'm not proud of this, but this appears to be a valid work-around. If you use this, please paste a big fat comment block explaining why you have to do this awful thing, and it should be removed as soon as this bug is fixed (either by ubuntu or vagrant...whatever the problem is). If you have a script provisioner, stick this bad boy in it:

if [[ ! -f /etc/init/fix-network-hack.conf ]]; then
   echo "#Fix vagrant-ubuntu-network-bug
      start on mounted MOUNTPOINT=/ and starting cloud-init-local

      task

      console output

      script
         sed -i '/post-up route del default dev/d' /etc/network/interfaces
      end script" > /etc/init/fix-network-hack.conf
fi

This will, before cloud-init is run, delete all occurrences of "post-up route del default dev" from /etc/network/interfaces. Then, after the network is set up, vagrant will put it back...but who cares, we'll just delete it again on start up.

You should probably make sure this hack is really what you want to do before you do it.

@trkoch
Copy link

@trkoch trkoch commented Jun 15, 2014

Any news on this? Cannot get config.vm.network "private_network", dhcp:true to work.

@trkoch
Copy link

@trkoch trkoch commented Jun 15, 2014

For the record, this affects both Trusty and Precise.

@trkoch
Copy link

@trkoch trkoch commented Jun 16, 2014

@mitchellh May I draw your attention to this issue and ask you to consider reopening it? Even if it is a bug in Ubuntu, at the very least I think it is an issue worth mentioning in the README. Especially considering that old versions of Ubuntu with this bug will be around for a while. I observed connection failures on several machines (20+) with freshly installed Virtual Box and Vagrant. As it stands, there is no way I know of to get the Ubuntu cloud images to work reliably in a DHCP setup (apart from crazy hacks as mentioned by @Houndie). Furthermore, can you please eleborate on why post-up route del default dev $IFACE is necessary?

@drpebcak
Copy link

@drpebcak drpebcak commented Jun 16, 2014

@trkoch I don't think it belongs in the README necessarily, but it would be nice if there was a more extended place where people could share their experiences and etc.

That was some of these nuances could be recorded in a friendlier way without keeping upstream issues open for Vagrant. I think the Mailing List scratches a bit of the itch for that, and there is also the IRC channel (which is a bit more ephemeral). Do you think there would be use for a forum or regularly updated extended documentation? Perhaps shifting some of the documentation load onto the community would be helpful in making these things friendlier to new users.

@Houndie
Copy link

@Houndie Houndie commented Jun 16, 2014

@trkoch Do we know if it's an upstream issue? I'm not doubting it, just as far as I know it hasn't really been investigated thoroughly yet.

I think the answer probably lies in the "post-up" line. If it's necessary (which, okay sure it might be..I have no idea what it does), then it's probably an ubuntu issue. If it's not then, well, it might be an ubuntu issue too, but one that vagrant can easily avoid by changing one line of code.

@trkoch
Copy link

@trkoch trkoch commented Jun 17, 2014

@drpebcak Granted it doesn't have to be the README, but I think it should be a place people where people notice. Perhaps Vagrant (on failure of any sort) could point to a Wiki page with general troubleshooting advice? Better yet, we fix this bug!

@Houndie We don't for sure. There are a lot of loosely related issues. I will try to dig into it.

@trkoch
Copy link

@trkoch trkoch commented Jun 23, 2014

Let's take a closer look at the interface configuration found in /etc/network/interfaces.

iface eth1 inet dhcp
    post-up route del default dev $IFACE

This basically means that after eth1 is brought up, the default route on eth1 should be removed. $IFACE is a variable that refers to the interface being configured and expands to eth1 in this example.

I deleted this line from my interface configuration, rebooted and ran this command manually (with root privileges).

$ route del default dev eth1
SIOCDELRT: No such process

As you can see, the command fails. I guess this is because there is no default route with the eth1 interface, as indicated by route. It turns out this command also fails if the machine was booted with the line in question.

post-up command
       Run command after bringing the interface up.   If  this  command
       fails then ifup aborts, refraining from marking the interface as
       configured (even though it has really been  configured),  prints
       an  error  message,  and exits with status 0.

In other words, if the post-up command fails, the interface remains unconfigured (no DHCP).

I still wonder what the purpose of that line might be? It was added in 33551ec ("Delete default routes for bridged networking on coming up"). From what I can tell this command is no longer necessary in recent versions of Ubuntu (I don't know about Debian).

A workaround is to make sure that the post up command never fails

post-up route del default dev $IFACE || true

With this line I can reboot my virtual machine without waiting 3 minutes.

@danielparks
Copy link
Contributor

@danielparks danielparks commented Jun 25, 2014

The post-up line is added in templates/guests/debian/network_dhcp.erb. When I remove that line, I no longer run into the problem.

Was this once necessary on some other Debian variant? If so, the || true change would be more backward compatible. Seems like the worst case is that route del… fails for some other reason and the machine boots without it.

@mitchellh this seems to be pretty clearly a Vagrant issue.

@danielparks
Copy link
Contributor

@danielparks danielparks commented Jun 25, 2014

I tried running the fixed version (without post-up) on Debian Squeeze (with private networking set to dhcp), and it works fine. Unsurprisingly, trying to delete eth1's default route results in an error:

$ sudo route del default dev eth1 ; echo $?
SIOCDELRT: No such process
7

Wild ass guess: was this to fix the case where eth1 came up before eth0 and established the wrong default route?

@mpereira
Copy link

@mpereira mpereira commented Jul 5, 2014

I've been experiencing this issue for a month, my "solution" has been destroying the vagrant machine and reprovisioning it.

@tritao
Copy link

@tritao tritao commented Jul 23, 2014

I'm also experiencing this issue with the Ubuntu Trusty image.

@michaelshobbs
Copy link

@michaelshobbs michaelshobbs commented Jul 28, 2014

bump. any movement on this? perhaps a PR would help expedite?

michaelshobbs pushed a commit to expa/buildstep that referenced this issue Jul 28, 2014
@trkoch
Copy link

@trkoch trkoch commented Jul 29, 2014

It seems to me that nobody really knows what this post-up command is meant to accomplish. A PR would fix this issue as far as I can tell, but could introduce another if we don't know what we are doing. I'm not sure if @mitchelhh is comfortable with appending '|| true' as described.

@michaelshobbs
Copy link

@michaelshobbs michaelshobbs commented Jul 29, 2014

In the meantime, I am using the following hack in my Vagrantfile when using the trusty64 box.

sed --in-place -e "s:post-up route del default dev \\$IFACE$:post-up route del default dev \\$IFACE || true:g" /etc/network/interfaces
@Jonasse79
Copy link

@Jonasse79 Jonasse79 commented Dec 11, 2015

Same issue. No real solution found.
UP!

@alexkart
Copy link

@alexkart alexkart commented Dec 11, 2015

@Jonasse79 did you try one of the solutions from my post above?

@Jonasse79
Copy link

@Jonasse79 Jonasse79 commented Dec 11, 2015

@alexkart Yep. First I checked if virtualization was enabled in the BIOS (OK) then I tried to "sudo" some specified instructions about VirtualBox GUI and SSH, then I tried to modify the vagrantfile...
My host OS is ubuntu 64 bits 14.04 (LTS) and I'm currently trying to launch Open Edx (cypress) with VirtualBox 5 and Vagrant 1.7.4 on a virtualised ubuntu 12 64 bits server OS.
I spent many hours reading posts but did not manage to solve my problem.

When I "vagrant up" everything is OK at launch, but then the connexion fails.
And then I got the "Trying to connect .. fail" message again and again!!!

@ahmedsaoudi
Copy link

@ahmedsaoudi ahmedsaoudi commented Dec 13, 2015

I have the same problem.
But I noticed that when I start a headless instance of Ubuntu Trusty, Vagrant fails at almost every attempt to launch it (passing occasionally, though). When using GUI, however, vagrant up works most (but not all) of the time.

Here's the guest's /interfaces.d/eth0.cfg:

auto eth0
iface eth0 inet dhcp

And its interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# Source interfaces
# Please check /etc/network/interfaces.d before changing this file
# as interfaces may have been defined in /etc/network/interfaces.d
# NOTE: the primary ethernet device is defined in
# /etc/network/interfaces.d/eth0
# See LP: #1262951
source /etc/network/interfaces.d/*.cfg

Anything wrong with those files?

@ddugovic
Copy link

@ddugovic ddugovic commented Dec 14, 2015

What operating systems does Vagrant's private_network actually work under? Clearly not Ubuntu...

@alexkart
Copy link

@alexkart alexkart commented Dec 14, 2015

try to login to your guest with vb.gui = true
and type ifconfig command, you should see eth0 network interface if you do not see it then, try

ifdown eth0
ifup eth0

if you see eth0 in ifconfig now and you can ssh to the guest, see this comment above #3860 (comment)

@ahmedsaoudi
Copy link

@ahmedsaoudi ahmedsaoudi commented Dec 14, 2015

@ddugovic I didn't setup any private_network inside my Vagrantfile. I use the NAT connection to access my guest machine which is, by the way a Windows 7 machine. But I'm curious: why (not 'what for') did you ask that question? I mean, what was in the files I pasted that seemed 'out of place' to you?

@ahmedsaoudi
Copy link

@ahmedsaoudi ahmedsaoudi commented Dec 14, 2015

@alexkart I just updated the said file and I'll see what will happen.

@ddugovic
Copy link

@ddugovic ddugovic commented Dec 14, 2015

@ahmedsaoudi I am responding to this comment and I have no problem with your Vagrantfile.

My needs are different since I need to use bridged mode for my guest which I need to be publicly accessible, and it sounds like bridged mode is completely broken (or not supported) for Vagrant.

@ahmedsaoudi
Copy link

@ahmedsaoudi ahmedsaoudi commented Dec 15, 2015

@ddugovic Oh! I'm sorry ^^'

@kumaxim
Copy link

@kumaxim kumaxim commented Dec 28, 2015

Hi everyone.
I solve the problem. My solution relevant for vagrant 1.7.4 and virtualbox 5.0.12

Root a problem - ubuntu 14.04(trusty, 32 bit) incorrect work with Intel network adapter. Use AMD network adapter to resolve the problem.

Example Vagrantfile this https://gist.github.com/Apach47/5277f9ac1ab4312b6ec5

Solution step by step

Step 1 - require

Add or change in the vagrantfile provider section:

config.vm.provider "virtualbox" do |vb|
    ### Change network card to PCnet-FAST III
    # For NAT adapter
    vb.customize ["modifyvm", :id, "--nictype1", "Am79C973"]
    # For host-only adapter
    vb.customize ["modifyvm", :id, "--nictype2", "Am79C973"]
  end

Step 2 - optional

  • Add auto_config: false to network section
  • Add follow code to provision section
config.vm.provision "shell", inline: <<-SHELL
    rm -f /etc/network/interfaces.d/eth1.cfg
    echo "auto eth1" >> /etc/network/interfaces.d/eth1.cfg
    echo "iface eth1 inet static" >> /etc/network/interfaces.d/eth1.cfg
    echo "address 192.168.35.25" >> /etc/network/interfaces.d/eth1.cfg
    echo "netmask 255.255.255.0" >> /etc/network/interfaces.d/eth1.cfg
    ifdown eth1 && ifup eth1
  SHELL

For default, vagrant change /etc/network/interfaces file and it is not a problem, but since ubuntu 14.04 setting for adapter store in interfaces.d folder. I suggest to follow this

@thisismyrobot
Copy link

@thisismyrobot thisismyrobot commented Jan 1, 2016

@Apach47 Thanks, that works perfectly!

I can confirm the fix for these same versions of vagrant and virtualbox.

@swaroopbekal
Copy link

@swaroopbekal swaroopbekal commented Jan 8, 2016

@Apach47 Thanks !

@vincentalvo
Copy link

@vincentalvo vincentalvo commented Feb 1, 2016

@Apach47 Amazing! Thanks it works !!!

@ghost
Copy link

@ghost ghost commented Feb 23, 2016

I had no problems with the private_network settings per se, but I had a similar problem, in which my VM would hang after the first reboot on the cloud-init screen, even after waiting out for 20m). It was cause by NFS mounts(the machine booted and rebooted fine when I commented out the NFS mounts from fstab) which I resolved by moving my mounts to rc.local.

This was with both ubuntu as host and guest btw.

@kenmorechalfant
Copy link

@kenmorechalfant kenmorechalfant commented Mar 8, 2016

@kumaxim Thanks, this appears to be working.

lc-thomasberger added a commit to adaptlearning/adapt_authoring that referenced this issue Mar 16, 2016
lookfwd added a commit to scalingexcellence/scrapybook that referenced this issue Sep 8, 2016
If you get

```
cloud-init-nonet[4.54]: waiting 10 seconds for network device
cloud-init-nonet[14.57]: waiting 120 seconds for network device
cloud-init-nonet[134.57]: gave up waiting for a network device.
```
Your netowrk card is not supported with the default adapter. See [here](hashicorp/vagrant#3860 (comment)). Try to uncomment the lines above.
@sgratzl
Copy link

@sgratzl sgratzl commented Dec 15, 2016

@kumaxim Thanks made my day.

It allows me that I can now run on my windows 10 pro pc both docker (hyper-v) and virtualbox via vagrant at the same time :)

@rfiedler
Copy link

@rfiedler rfiedler commented Jan 18, 2017

In my case only config the my private_network :

config.vm.network :private_network, ip: "192.168.33.10"

@kaoz70
Copy link

@kaoz70 kaoz70 commented Mar 1, 2017

This seems to only happen when using:
config.vm.network "forwarded_port", guest: 80, host: 8080

If you only use:
config.vm.network :private_network, ip: "192.168.33.10"

It wont happen

@mvdinter
Copy link

@mvdinter mvdinter commented Jul 16, 2017

@kumaxim thanks!! Step 1 did it form me to get an 32bits Ubuntu working in combination with Vagrant 1.9.7 and VirtualBox 5.1.22r115126.

lonre added a commit to lonre/homeland that referenced this issue Aug 4, 2017
doralteres added a commit to doralteres/panda that referenced this issue Nov 8, 2017
@hashibot
Copy link

@hashibot hashibot bot commented Apr 1, 2020

I'm going to lock this issue because it has been closed for 30 days . This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@hashibot hashibot bot locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet