Support private topology deployment behind http proxy

[DerekV]

Currently, deploying to a private network without NAT, but with an available http proxy, will fail at a number of points.
I have coerced it to work with some light code modifications to thread environment variables and configuration files through to all the right places. I will get the code linked here once I get it cleaned up (get out hardcoded urls etc), for reference and further review.

Meanwhile the question is, does kops want to take on the goal of supporting deployment behind http_proxies? Even though we have run this way for a couple years now with only minor annoyance, I have realized a number of implications about this configuration and supporting it while trying to get it to work with kops. Here's the rub: to really do it right, I believe that some discipline will need to be maintained around ensure support is propagated correctly to everywhere http calls (including, for example, AWS or other provider API calls).

Generally, the problem is that http_proxy support must be implemented by every http client, and there is no universal, standard way to inject http_proxy configuration. Go net/http will respect http_proxy, https_proxy, and no_proxy environment variables in more or less the same way that wget and curl interpret them, and this is how I was able to get it to work. The problem is that, if you just set http_proxy, http clients will try to forward every single http request to the proxy, even requests for things located in the same private subnet as Kubernetes, on the overlay network itself, or on some other private IP address. It does this regardless if the http request uses a hostname or an IP address.

The no_proxy environment variable allows you to exclude hostnames and hostname suffixes. It matches on hostname suffix only, eg shoes.example.com,.mycorp.com,.mynet.local would exclude privateserver.mynet.local. So really the core issue behind the core issue is that for some reason, that never seemed weird to me until now, is that ip addresses are expressed most significant "domain" first, but hostnames express them last, and the no_proxy convention assumes only hostnames. It has to provision to handle ip addresses, where the prefix is the most significant, nor does it understand ranges, globs, nor CIDR notation. This holds for curl, wget, and by extension, go. I'm guessing getting all three to change would not be an easy mountain to climb. I have been able to hack around this by exuding all ip address urls by including every ending ip in no_proxy .0,.1,.2 ... .255, but this feels like a giant hack, is likely fragile (someone may need to address an external server by IP address at some point, which would then break this hack).

So environment variables might not be the best answer ultimately. Perhaps maintaining a light wrapper around net/http

Here is a rough list of places I modified in the process of getting it to work:

• set the environment variables within cloud-init itself
• added environment variables to /etc/environment (can be done in cloud-init)
• added environment variables to /etc/default/docker for docker daemon, can be done in cloud-init (can be done in cloud-init)
• added proxy config to apt.conf.d, ala echo "Acquire::http::Proxy \"http://${PROXY}\";" > /etc/apt/apt.conf.d/30proxy (can be done in cloud-init)
• added environment variables to systemctl "DefaultEnvironment" ala echo DefaultEnvironment=http_proxy=${PROXY} https_proxy=${PROXY} ftp_proxy=${PROXY} no_proxy=${NOPROXY} >> /etc/systemd/system.conf (can be done in cloud-init)
• modify nodeup/pkg/model/protokube.go to pass along environment variables (next to the s3 environment variables already being passed here
• modify manifests for dns-controller and kube-dns addons to pass environment variables
• modify manifests for kube-api and kube-controller manager to pass environment variables

[DerekV]

It has been mentioned that supporting self-hosting of resources (#715) might help fill the same need. I agree, that it partly does. In the context of an AWS VPC, in fact, I was able to get at the resources hosted on s3 by adding a VPC Endpoint for s3 and allowing access to the relevant buckets. Unfortunately, the only AWS service that provides an VPC Endpoint connection is s3 (presumably there will be others), but there is no predicting when an endpoint for API calls might be added. And I suppose in the general case, there may be other situations where the API provider needed by services is outside the private network.

[DerekV]

Only one known sticking point in Kubernetes itself already had an issue open here: kubernetes/kubernetes#35186 . This is the one where I needed the ip address no_proxy hack. Everything else seemed to use hostnames or just address the API server at one specific IP address, namely 100.64.0.1

[chrislovecnm]

@DerekV ping me when you have a chance, this may be coming on my plate. What about cloud provider and such? Are you putting aws API calls through a proxy

[johnzeringue]

We've gone through an exercise similar to @DerekV.

What about cloud provider and such? Are you putting aws API calls through a proxy

AWS API calls have to go through the proxy.

We haven't built a proper API in front of our solution (we use environment variables instead of kops configuration/flags), but we'll open a pull request to share our progress until it's finished.

[DerekV]

@johnzeringue I've got a PR that I am working on

[DerekV]

Sorry I had my head down trying to get our cluster into production. I should have responded to @chrislovecnm publicly.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[chrislovecnm]

Support is in for this