Can't target bosh-lite director

[zmb3]

I am unable to target the bosh-lite director from my macOS host.

❯ bosh target 192.168.50.4 lite
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
[WARNING] cannot access director, trying 4 more times...
[WARNING] cannot access director, trying 3 more times...
^C
Exiting...

I have run the add-route script and I am able to ping both 192.168.50.1 and 192.168.50.4.
I also installed the extension pack as suggested in #202.
I tried restarting the VM several times as suggested in the troubleshooting, but the error persists.

I can target the director from inside the bosh-lite VM, so I know it's running and this is just a network issue.

My environment:

❯ vagrant --version
Vagrant 1.8.6
❯ VBoxManage --version
5.1.6r110634

VBox adapter from my macOS host:

vboxnet2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    ether 0a:00:27:00:00:02
    inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255

Network inside the VM:

vagrant@agent-id-bosh-0:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:16:0d:c1
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe16:dc1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:561 errors:0 dropped:0 overruns:0 frame:0
          TX packets:399 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:53063 (53.0 KB)  TX bytes:49860 (49.8 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:85:e2:59
          inet addr:192.168.50.4  Bcast:192.168.50.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe85:e259/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:58 errors:0 dropped:0 overruns:0 frame:0
          TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4964 (4.9 KB)  TX bytes:4122 (4.1 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:65151 errors:0 dropped:0 overruns:0 frame:0
          TX packets:65151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8946460 (8.9 MB)  TX bytes:8946460 (8.9 MB)

lo:1      Link encap:Local Loopback
          inet addr:10.254.50.4  Mask:255.255.255.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1

I also tried setting the gateway to 192.168.50.1 instead of 192.168.50.4, but it didn't change the result.

[dpb587-pivotal]

If you run netstat -rn, are you able to verify the correct routes were added and that you don't have any conflicting routes?

[zmb3]

How does this look?

Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.1        UGSc          579        0     en0
10/16              192.168.50.1       UGSc            1        0 vboxnet
10                 10.35.59.249       UGSc            5        0   utun1
10.35.59.1/32      10.35.59.249       UGSc            1        0   utun1
10.35.59.249       10.35.59.250       UH             18        0   utun1
10.128/9           10.35.59.249       UGSc            2        0   utun1
10.244/16          192.168.50.4       UGSc            1        0 vboxnet
66.116.126.128/28  10.35.59.249       UGSc            1        0   utun1
67.214.213/24      10.35.59.249       UGSc            1        0   utun1
67.214.217/24      10.35.59.249       UGSc            1        0   utun1
67.214.219/24      10.35.59.249       UGSc            1        0   utun1
67.214.220/23      10.35.59.249       UGSc            1        0   utun1
67.214.222/24      10.35.59.249       UGSc            1        0   utun1
127                127.0.0.1          UCS             2       18     lo0
127.0.0.1          127.0.0.1          UH              7   157169     lo0
127.0.53.53        127.0.0.1          UHWIi           1        4     lo0
172.16/22          10.35.59.249       UGSc            1        0   utun1
172.16.100/22      10.35.59.249       UGSc            1        0   utun1
184.169.136.236/32 10.35.59.249       UGSc            1        0   utun1
192.168.1          link#4             UCS             7        0     en0
192.168.1.1/32     link#4             UCS             2        0     en0
192.168.1.1        30:b5:c2:8:7e:e    UHLWIir       580     4106     en0    771
192.168.1.100/32   link#4             UCS             1        0     en0
192.168.1.101      a4:77:33:b9:52:14  UHLWIi          3    40281     en0    330
192.168.1.102      link#4             UHLWIi          1        0     en0
192.168.1.103      link#4             UHLWIi          1        2     en0
192.168.1.104      88:28:b3:3f:c1:53  UHLWIi          1       49     en0    134
192.168.1.105      d0:e7:82:c6:70:2e  UHLWIi          3    30028     en0    330
192.168.1.255      link#4             UHLWbI          1        9     en0
192.168.50         link#18            UC              3        0 vboxnet
192.168.50.1       a:0:27:0:0:2       UHLWIi          3       18     lo0
192.168.50.4       link#18            UHLWIi          2      234 vboxnet
224.0.0/4          link#4             UmCS            3        0     en0
224.0.0.251        1:0:5e:0:0:fb      UHmLWI          1        0     en0
239.255.255.250    1:0:5e:7f:ff:fa    UHmLWI          1     3917     en0
255.255.255.255/32 link#4             UCS             1        0     en0

[dpb587-pivotal]

Routes looks sane.

Are you able to verify port connectivity (nc -z 192.168.50.4 25555)? If so, routes are probably correct.

If you're on a newer bosh-lite, have you tried passing the CA certificate via bosh --ca-cert ca/certs/ca.crt target 192.168.50.4 lite?

[zmb3]

Looks like the routes are okay:

nc -z 192.168.50.4 25555
Connection to 192.168.50.4 port 25555 [tcp/*] succeeded!

No-go with the CA cert:

bosh --ca-cert ca/certs/ca.crt target 192.168.50.4 lite
[WARNING] cannot access director, trying 4 more times...
[WARNING] cannot access director, trying 3 more times...

I am on the latest bosh-lite - just cloned it yesterday.

git rev-parse HEAD
566dbe0bd7d203c3ee66ae3afd70d24cc7f1058a

[dpb587-pivotal]

Do either of the following provide any more details?

# with a simple connection
$ curl -vk https://192.168.50.4:25555/info

# with the certificate
$ curl -v --cacert ca/certs/ca.crt https://192.168.50.4:25555/info

[zmb3]

I am able to hit the info endpoint with both of those commands:

❯ curl -vk https://192.168.50.4:25555/info
*   Trying 192.168.50.4...
* Connected to 192.168.50.4 (192.168.50.4) port 25555 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: 192.168.50.4
> GET /info HTTP/1.1
> Host: 192.168.50.4:25555
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Fri, 14 Oct 2016 19:18:43 GMT
< Content-Type: application/json
< Content-Length: 358
< Connection: keep-alive
< X-Content-Type-Options: nosniff
<
* Connection #0 to host 192.168.50.4 left intact
{"name":"Bosh Lite Director","uuid":"fb70f16a-4d6f-41f8-bc2e-0181a03dff18","version":"1.3262.3.0 (00000000)","user":null,"cpi":"warden_cpi","user_authentication":{"type":"basic","options":{}},"features":{"dns":{"status":false,"extras":{"domain_name":null}},"compiled_package_cache":{"status":true,"extras":{"provider":"local"}},"snapshots":{"status":false}}}

[dpb587-pivotal]

So... why isn't it working? :)

Do you maybe have some outdated settings in your ~/.bosh_config or something? If you try BOSH_CONFIG=/dev/null bosh -t 192.168.50.4 status, does it at least connect? If so, perhaps try moving ~/.bosh_config out of the way and trying from scratch?

[zmb3]

I wish I knew! Still no luck:

❯ BOSH_CONFIG=/dev/null bosh -t 192.168.50.4 status
Config
             /dev/null

Director
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
[WARNING] cannot access director, trying 4 more times...
[WARNING] cannot access director, trying 3 more times...
^C
Exiting...

Same result after moving ~/.bosh_config out of the way too:

bosh target 192.168.50.4 lite
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
[WARNING] cannot access director, trying 4 more times...
[WARNING] cannot access director, trying 3 more times...
^C
Exiting...

[dpb587-pivotal]

I wonder if there are issues with ruby on macOS Sierra when trying to compile against openssl. Are you using the system ruby? Could you try using rbenv or chruby and see if that builds/runs any differently?

[dpb587-pivotal]

Just tried on a dev Sierra machine and it seemed to work fine, but I don't think we're using the system ruby and I think we did some separate installations of openssl with brew for other build issues.

[zmb3]

I reinstalled the bosh CLI, and saw a different message this time:

❯ bosh target 192.168.50.4 lite
Ignoring eventmachine-1.0.9.1 because its extensions are not built.  Try: gem pristine eventmachine --version 1.0.9.1
Ignoring http_parser.rb-0.6.0 because its extensions are not built.  Try: gem pristine http_parser.rb --version 0.6.0
[WARNING] cannot access director, trying 4 more times...
Target set to 'Bosh Lite Director'

I ran the gem pristine commands that it suggested and now I'm able to log in just fine. Thanks for the help!

[clingmann]

I'm also on Sierra 10.12.1 and I have the exact same symptoms and haven't been able to get it to run with rebuild or vagrant or reinstall of the packages. despite being able to SSH in to the VM fine. I've gone through the pre-reqs and have Xcode and cli installed and the gem packages say they installed successfully. Could you provide a bit more detail about how you got it to work by chance?

[zmb3]

@clingmann I'm not a ruby guy, so forgive me if I'm way off, but I think when I first installed the bosh CLI the only ruby I had was the system ruby. At some point after that I installed ruby through homebrew. My guess is that when I removed the bosh CLI and installed it for the second time it picked up the newer Ruby.