CloudFoundry CLI should support SOCKS5 proxies

[jkugler]

Command

cf

CLI Version

cf version 6.23.0+c7866be18-2016-12-22

CC API Endpoint Version

N/A

It would be very nice if the CF cli supported the usage of socks proxies. We do a lot of tunneling with SSH SOCKS proxies (the -D option). This allows us to communicate with various environments without having to be on that network directly, and not having to set up full-blown HTTP(S) proxies for all the environments.

Being able to use SOCKS5 proxies directly from the CF cli would be VERY convenient.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/137058285

The labels on this github issue will be updated when the story is started.

[dkoper]

Hi @jkugler

Thanks for this feature request. I see it already received a few thumbs up!
We'll do some research.

Regards,
Dies Koper
CF CLI PM

[dkoper]

Hi @jkugler

all-proxy is an environment variable some tools, e.g. Linux curl, accept. If the cf CLI would accept this, so you could e.g. do a all_proxy=socks5://user:pwd@127.0.0.1:1080 cf api, would that satisfy your requirement?
There may be complications for cf commands that make connections other than http(s), e.g. cf ssh. Is that a blocker?

[jkugler]

To use a Socks proxy, you can use add this to an ssh invocation:

-o "ProxyCommand nc -X 5 -x <socks host>:<socks port> %h %p"

That will, of course, not work for systems without netcat installed (windows). Also, if the cf binary is not invoking an external SSH client, of course that won't work. If it is an internal library call, I would think that library would have a way to use a Socks proxy. Only being able to use it for HTTP calls would, I would think, be very frustrating when a command was invoked that required an SSH connection. I've only begun using the cf cli, so I really don't know how often we would need command invoked over SSH.

[dkoper]

The cf CLI does not invoke an external SSH client.
The cf ssh command opens an SSH shell to your app instance's container. This is the only command that uses SSH, all others are HTTP (or websocket, in case of cf logs and other commands that display logs).

So basically, what I'd like to confirm when you say "We do a lot of tunneling with SSH SOCKS proxies (the -D option). This allows us to communicate with various environments without having to be on that network directly", is whether you mean you need a SOCKS proxy for the CLI to connect to the Cloud Foundry APIs because your CF is installed on those environments, or need it for other purposes (incl. accessing your app's container with ssh).

[jkugler]

So, still trying to understand. How does it open an ssh shell to the app's container if it doesn't invoke an external SSH client? Is it an SSH client written via a go library built in to the cf cli? And if that is the case, can that library use a socks proxy?

[jkugler]

So...comes down to: if an ssh connection is being invoked for the connection to the container, then it would need to pass through the socks proxy as well. Is there a way to utilize the socks proxy information in the ssh connection as well?

[Akhena]

This issue is also blocking for us. We work in a corporate environnement where all outgoing internet traffic has to go through a proxy.
Everything works fine with the "cf" commands EXCEPT for "cf SSH" that seems to ignore our rpoxy configuration :(

This is really a pita, we don't have any workaround at the moment...If anyone has any kind of workaround, I would be eternally grateful to them.

[cppforlife]

@dkoper bosh cli v2 started supporting socks5 recently as well. it uses BOSH_ALL_PROXY env var instead of all_proxy as i found it pretty annoying to not be able to use same terminal session easily for proxied and non-proxied requests. users that do not have to make such distinction can of course just set BOSH_ALL_PROXY=$all_proxy.

re ssh wrapping: crypto/ssh library allows to pass in custom Conn which could have been created via a custom dialer that respects socks proxy configuration. here is an example: https://github.com/cloudfoundry/bosh-cli/blob/bd274dde4edba031cfcf7982c2e53a9f393aefb9/ssh/client.go#L116.

[dkoper]

@cppforlife Thanks for chiming in!
When you talk about proxied and non-proxied requests from the same terminal, are you referring to switching between BOSH Directors that do require SOCKS and those that don't? Or the issue that all_proxy gets picked up by other commonly used Linux tools, like curl, which you use e.g. for downloading stemcells and artfacts from bosh.io, in which case it generally does not need the SOCKS proxy?

@Akhena It's not that cf ssh ignores your proxy configuration, it's that cf only supports HTTP proxies while ssh makes non-HTTP calls. SOCKS proxy support is the answer to that, but you'll need to have a SOCKS proxy server allowing outgoing Internet traffic.

[dkoper]

Sorry - didn't mean to close this.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/138101297

The labels on this github issue will be updated when the story is started.

[Akhena]

@dkoper Isn't it possible that cf ssh would also support a classic HTTP proxy (by using HTTP CONNECT)?
It seems to me that it's what other ssh clients (like Bitvise for instance) use to be able to tunnel their ssh connections.

[jkugler]

Just chiming in again. While @Akhena's idea is a good one, in our case we do NOT have HTTP proxies in to these environments. The socks proxy created by our ssh -D connection is the only one we have, so using HTTP proxies with CONNECT would not work for our SSH connections.

[Akhena]

I agree @jkugler , my suggestion is made at the wrong place.
Ideally, I should open a new issue asking for "Support for HTTP proxies".

Actually, in our corporate environnement, adding CF cli support for any of the two types of proxies (HTTP / SOCKS5) would solve our problem.

[dkoper]

@Akhena I believe SOCK5 procies are more common, so we'll explore that first.

[jfmyers9]

@jkugler Are you aware that you can use an external SSH client to connect to applications? I recently verified that using the normal OpenSSH client, the -D flag appears to work just fine. You can find documentation on how to SSH to CF applications in this documentation. There is still a conversation around whether or not the CF CLI should support this functionality, but hopefully this will provide a decent work around for the time being.

[drnic]

I think this is now implemented for free when we moved to go1.9

golang/go@36f55a8

Quick sanity check worked in my new env. Neat.

export https_proxy=socks5://localhost:9999

[dkoper]

@jkugler, @drnic, all

So since cf CLI 6.31.0 you should be able to set your SOCK5 proxy with https_proxy and most cf CLI commands work.
However, we've found two commands so far that do not work: logs (without the --recent option) and ssh). We're still investigating logs, but it seems ssh doesn't work because Golang 1.9's SOCKS5 support was not added to the APIs used by its ssh library.

Once we figure out why logs is not working, I'll look at updating the docs about this feature (and its limitations).

[jkugler]

@dkoper thanks! In our use case, we configured our ssh client to use the socks proxy directly, so never used cf ssh, so probably wouldn't have hit that. What is causing the logs failure? Does it hit the API in a different way? Is there a way to make sure it uses and http(s) connection that gets routed through the proxy? It would seem like if logs used a proxy before, it would hit the same code path in Go that caused the go runtime to pick up the socks proxy...but I have no knowledge of the logs call, so... :)

[dkoper]

@jkugler We hope to answer the why of the failure in the coming days.

What I see is that cf logs APP -v makes a number of API calls successfully (you can try yourself) but then hangs when it tries to switch protocols to upgrade to a websocket connection. (It is a streaming API that doesn't use plain HTTP/S).
This fails for an HTTP proxy as well, but I would have expected it could work for SOCKS5.

[jkugler]

@dkoper Thanks for the update! Protocol is NOT my area of expertise. I'm sure you can google as well as I can, but I came across this and thought it might be helpful. https://www.infoq.com/articles/Web-Sockets-Proxy-Servers tl;dr: sometimes websockets works with proxies...sometimes it does not. Can depend on the proxy. :(

[dkoper]

@jkugler We believe noaa, the library we use to talk to loggregator, needs changes to support this. Created cloudfoundry/noaa#38.

[dkoper]

@jkugler I'd like to document the SOCKS5 proxy support (with its current limitations with ssh and logs) as I assume it's mostly usable. Do you have a suggestion for where to document it? Where would you look for it?

We have https://docs.cloudfoundry.org/cf-cli/http-proxy.html. I can't imagine SOCKS5 proxies are used widely enough to warrant a chapter of its own. I'm thinking of updating this chapter's title to include SOCKS5 and include it here, assuming HTTP and SOCKS5 proxy support are often documented together so that's where people would expect to find it. Does that sound fair?

Of course the search DB will be updated so searches for SOCKS5 should work no matter where we document it.

[abbyachau]

Closing this issue as we will release cf cli v6.38.0 soon which will include this feature. Please note support for SOCKS5 will only be available via the cf v3-ssh command, which will be noted in the release notes. If any feedback/additional comments, feel free to reopen or create a new issue. Thanks again.

[jkugler]

@abbyachau Are you saying the use of socks proxies will not be available for HTTPS commands as this ticket originally requested?

[abbyachau]

Hey @jkugler we've implemented support for SOCKS5 for cf v3-ssh. This has yet to be released but we will publish CF CLI v6.38.0 soon, which will include this feature. As this is a long thread, summarising here:

• CF CLI 6.33.1 includes SOCKS5 support for logs - release notes

• CF CLI 6.38.0 yet to be released but soon; includes SOCKS5 support for cf v3-ssh - see release notes when 6.38.0 comes out

Let me know if you have any further questions/comments.

[jkugler]

Thanks answering, but sadly it still doesn't answer my question. :)

This issue was opened because I wanted to be able to set an environment variable such as

export https_proxy=socks5://localhost:9999

or

all_proxy=socks5://user:pwd@127.0.0.1:1080

(See this comment: #1047 (comment))

and that would direct all CF API calls (which are http or https traffic) over the given socks proxy. Will the feature you reference in 6.38.0 do that? Or will it just support ssh over socks proxy?

[abbyachau]

Hey @jkugler we will support ALL_PROXY=socks5://socks_proxy.example.com and this will be updated in the official CF documentation which we will link to in the release notes. Let me know if you need anything else.

[jkugler]

Very cool. Thanks!

[johnlonganecker]

I am having a hard time to get cf logs to work while using the https_proxy set to a socks proxy on my windows box. Since cf logs goes over wss:// is there any talks as to when it will be supported as well?