Bosh worker node cannot cf login via CF CLI

[leungandrew]

Hello,

Our office is trying to deploy Concourse via Cluster Bosh on AWS. Following the tutorial, we were successfully able to get Concourse up and running, however the last step in order for us to use Concourse is for the cf-resource to properly deploy our applications to PWS.

Unfortunately, after many attempts, it appears that the worker node cannot seem to cf login for some reason. We had originally thought there were networking issues from AWS side, however we are able to curl all the endpoints and get responses.

curl 'https://api.run.pivotal.io'
{
  "code": 10000,
  "description": "Unknown request",
  "error_code": "CF-NotFound"
}

curl -v 'https://login.run.pivotal.io'
* Rebuilt URL to: https://login.run.pivotal.io/
* Hostname was NOT found in DNS cache
*   Trying 52.200.161.90...
* Connected to login.run.pivotal.io (52.200.161.90) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*    subject: C=US; ST=California; L=San Mateo; O=Pivotal Software, Inc.; CN=*.run.pivotal.io
*    start date: 2015-01-23 00:00:00 GMT
*    expire date: 2017-01-27 12:00:00 GMT
*    subjectAltName: login.run.pivotal.io matched
*    issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*    SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: login.run.pivotal.io
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Date: Thu, 21 Jul 2016 07:07:44 GMT
< Expires: 0
< Location: https://login.run.pivotal.io/login
< Pragma: no-cache
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Vcap-Request-Id: 1e7a8cbf-6e22-41ed-532b-97d93ed102a7
< X-Xss-Protection: 1; mode=block
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host login.run.pivotal.io left intact

We can telnet:

telnet api.run.pivotal.io 80
Trying 52.200.161.90...
Connected to api.run.pivotal.io.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

telnet api.run.pivotal.io 443
Trying 52.72.174.130...
Connected to api.run.pivotal.io.
Escape character is '^]'.
^]

telnet> quit
Connection closed.

We have attempted to roll back to older versions of the CLI, pulling the code and compiling it directly, all resulting to the same error.

Command

cf login

CLI Version

6.20.0+25b1961 / linux

Error

FAILED
Error performing request: Get http://api.run.pivotal.io/v2/info: dial tcp: i/o timeout
TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.

Stack Trace

CF_TRACE=true ./cf login

API endpoint> https://api.run.pivotal.io

REQUEST: [2016-07-21T07:14:52Z]
GET /v2/info HTTP/1.1
Host: api.run.pivotal.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.20.0+25b1961 / linux


FAILED
Error performing request: Get https://api.run.pivotal.io/v2/info: dial tcp: i/o timeout
TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.

CC API Endpoint Version

CF_TRACE=true ./cf api https://api.run.pivotal.io
Setting api endpoint to https://api.run.pivotal.io...

REQUEST: [2016-07-21T07:14:04Z]
GET /v2/info HTTP/1.1
Host: api.run.pivotal.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.20.0+25b1961 / linux


FAILED
Error performing request: Get https://api.run.pivotal.io/v2/info: dial tcp: i/o timeout
TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.

CF Trace

See above

Platform Details

Ubuntu Trusty AWS Xen-HVM Light via Bosh Stemcell:

https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/light-bosh-stemcell-3262.2-aws-xen-hvm-ubuntu-trusty-go_agent.tgz

Shell

Linux Bash

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/126765001

The labels on this github issue will be updated when the story is started.

[dkoper]

Hi @xtreme-andleung

Can you try CF_DIAL_TIMEOUT=15 cf login? The timeout defaults to 5 sec, so cranking it up to 15 sec and try may tell us more.

Regards,
Dies Koper
CF CLI PM

[leungandrew]

@dkoper Thanks Dies! Looks like that solved our problem. The CLI help screen never provided such an env variable. Would've saved us a many hours of investigating network connectivity issues with the node. Perhaps the latency from an Australian server is too high and the response is taking much longer to return.

Is this env var something that should be added to the help screen?

Thanks,
Andrew

[dkoper]

@xtreme-andleung Thanks for that feedback!
There are many timeouts we can make configurable, confusing people just as much, so I didn't want to add it to the help screen until I got some user feedback that this particular timeout actually solved someone's issue. You're one of the first to reply!
I'll look at adding it to the help page.
I'm in Sydney and have no problems connecting to that server, even from our slow corporate Internet/http proxy. Note that this timeout controls the time it takes to establish the connection (including the resolution of the domain name), not the time it takes for the HTTP response to return.
Maybe you have an issue with a slow DNS server?

[dkoper]

Adding the env var to the help page will be covered in this story:
https://www.pivotaltracker.com/story/show/126948021