StackStorm on Vagrant | https://10.10.10.10 does not respond

[mlemartien]

Hi,

I'm trying to play with st2 using Vagrant and following the instructions. The image gets loaded and the VM nicely boots. I can ssh onto it but when I try to run the portal from https://10.10.10.10 it does not respond. Could my VirtualBox config have anything to do with this? I have not changed the default settings I reckon.

Thanks a ton in advance.

[arm4b]

@mlemartien Can you please share more info about your host OS (Linux/Win/Mac), Vagrant version and Virtualbox version?

Try to find if network interface is created via Virtualbox. Here is how Virtualbox Host Network Manager looks for me:

Do you see something similar under your Virtualbox networks?

[mlemartien]

Thanks for your prompt reply. When I go to the Host Network Manager screen in Global Tools, the list is empty (which is weird I assume).

However when I run ifconfig -a inside the Vagrant VM, I get the following, which seems correct to me):

enp0s3    Link encap:Ethernet  HWaddr 08:00:27:88:8c:b8
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe88:8cb8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:756 errors:0 dropped:0 overruns:0 frame:0
          TX packets:538 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:86392 (86.3 KB)  TX bytes:79002 (79.0 KB)

enp0s8    Link encap:Ethernet  HWaddr 08:00:27:08:5d:3a
          inet addr:10.10.10.10  Bcast:255.255.255.255  Mask:255.255.255.254
          inet6 addr: fe80::a00:27ff:fe08:5d3a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:92667 errors:0 dropped:0 overruns:0 frame:0
          TX packets:92667 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:31336089 (31.3 MB)  TX bytes:31336089 (31.3 MB)

[arm4b]

Yes, something is blocking the creation of network interface on your host machine.

Can you do vagrant destroy --force and then share the output of

VAGRANT_LOG=info vagrant up

command?

Can put the output in https://gist.github.com for better readability.

[mlemartien]

When looking at the logs I saw that vagrant was struggling to connect to certain VBox services so I rebooted my computer. Now the interface is created but I still can not connect to https://10.10.10.10

Here's the output of the vagrant log. Thanks again for your help, much appreciated:
https://gist.github.com/mlemartien/74f67cdfb637ef14f56ccbb1d2dcc06d

[arm4b]

Alright, that's better!

Try to ping 10.10.10.11 and ping 10.10.10.10 within your both host machine and guest VM, - check if they both can respond.

[mlemartien]

From the host machine: 10.10.10.10 responds well, 10.10.10.11 does not:

$ ping 10.10.10.11
PING 10.10.10.11 (10.10.10.11): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
Request timeout for icmp_seq 0
ping: sendto: Can't assign requested address
Request timeout for icmp_seq 1
ping: sendto: Can't assign requested address
Request timeout for icmp_seq 2
ping: sendto: Can't assign requested address
Request timeout for icmp_seq 3
^C
--- 10.10.10.11 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

Same from the guest, but the ping output is not the same for 10.10.10.11. It gets stuck like this:

vagrant@stackstorm:~$ ping 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data.

And when I interrupt it:

vagrant@stackstorm:~$ ping 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data.
^C
--- 10.10.10.11 ping statistics ---
77 packets transmitted, 0 received, 100% packet loss, time 76204ms

[arm4b]

Is this result from the host machine or guest machine?
I mentioned to try both.

[mlemartien]

First part is from the host, second part from the guest. I think it is mentioned in my post

[mlemartien]

Also, curk -k https://10.10.10.10 responds well from inside the guest. Doing the same from the host returns:
curl: (7) Couldn't connect to server

Could that not have something to do with forwarding port 80?

[arm4b]

Alright, NP. I see now the post is updated.
Interesting that 10.10.10.10 at least can respond from the host machine.

Try next ssh vagrant@10.10.10.10 from the host to see if connection could be established.

[mlemartien]

Nice idea. It returns:

$ ssh vagrant@10.10.10.10 -vvv
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.10.10.10 port 22.
ssh: connect to host 10.10.10.10 port 22: Permission denied

[arm4b]

Alright, seems "private networking" is messed up for some reason in Virtualbox for your host machine.
The fall-back is to rely on port forwarding which is safer and might have less conflicts, comparing to creating network + private IP for the VM.
We've discussed it previously here: StackStorm/packer-st2#29

Let me add that as optional way to access the st2web. I'll let you know soon when the new box is available.

In a meantime, you can destroy the VM (vagrant destroy --force), remove the vboxnet0 from the Virtualbox manually, restart the host machine and try vagrant up again after reboot.
Considering that something like that partially worked previously, - it may help.

[arm4b]

@mlemartien The new Vagrant box is deployed.
Please follow the instructions to update to the new box: https://docs.stackstorm.com/install/vagrant.html#updating-the-vagrant-box

After vagrant up, you can access StackStorm Web UI via https://127.0.0.1:9000/ as an alternative method.
Let me know if that worked for you.

We'll keep port forwarding as experimental fallback option for now and include it in docs if more Virtualbox issues with private networking will appear in future.

[mlemartien]

Thanks, that does the trick!

[LindsayHill]

I hit the same issue with my setup. Something screwy with Virtualbox networking. Ping from host to 10.10.10.10 works, but https://10.10.10.10/ does not.

Using https://127.0.0.1:9000/ works, but this is not currently documented.

My first guess would be that it is something to do with the use of a /31 mask. I am using Virtualbox 5.2.12 on macOS 10.13.5.

[arm4b]

@LindsayHill The "base" Vagrantfile (this one: https://github.com/StackStorm/packer-st2/blob/master/Vagrantfile.template) is located in ~/.vagrant.d/boxes/stackstorm-VAGRANTSLASH-st2/2.7.2-20180523/virtualbox/Vagrantfile (not sure about your env, but should be a similar dir).

You can try to play with the networking there and verify if it's mask-related.

With your skills it's definitely would be great to find the root cause and what is the blocker there.
Random ideas to look at: Virtualbox network manager, overlapping networks, existing vbox interfaces, subnets, routes, reboot (like https://stackoverflow.com/a/28337424/4533625), etc.

[LindsayHill]

I did some more investigation. This is definitely related to the use of a /31 subnet mask. It looks like macOS doesn't like /31 (and I'm not surprised, the use of /31 is a bit of a hack, mainly only used for router<->router P2P connections).

I fixed it by changing my ~/.vagrant.d/boxes/stackstorm-VAGRANTSLASH-st2/2.7.2-20180523/virtualbox/Vagrantfile to use 255.255.255.252, and manually removing the previously created 10.10.10.10/31 network.

[arm4b]

Awesome! 👍

[mlemartien]

That's great feedback, thanks!

[arm4b]

The fixed in StackStorm/packer-st2#41 by @LindsayHill box is deployed to Vagrant cloud.

You can update the environment, follow the docs: https://docs.stackstorm.com/install/vagrant.html#updating-the-vagrant-box