Use http_proxy only for cloud provider interaction

[MatthiasWinzeler]

/kind feature

We're using k8s on VMWare and use the VSphere cloud provider to provision dynamic volumes.
However, in our enterprise setup, we need to access the VSphere API over a HTTP forward proxy.
This forward proxy only allows access to VSphere/ESX hosts and no other endpoints (i.e. no internet).

We achieved this by exporting the env var http_proxy=http://user:password@our-forward-proxy for all kube binaries that interact with the cloud provider (kubernetes-api, kubernetes-controller-manager, kubelet).

Since we can only set the http_proxy on process/binary level, this means that all HTTP interaction of the above-mentioned binaries go over this proxy - even cluster-internal traffic, namely traffic from the api to the kubelets, from the kubelets to the api and from the kubelet to the overlay networks of the pod (for healthchecks etc.). Also, all HTTP requests of these binaries that target the external world (are there any? i.e. for registries?) would go over the proxy and not find their target.

We worked around this by putting all known endpoints that should not go over the proxy in no_proxy. Currently, the k8s node network and the pod overlay network are excluded from the proxy.

With this extensive no_proxy, the cluster seems to work, however we're not feeling comfortable with this solution: We're not sure we did not forget to whitelist something or something will be added in the future that will break. Also, if we discover some external endpoints that kube would like to talk to, we would have to add these to no_proxy, too. So we would prefer it if we have a way to only specify the proxy where it is needed, i.e. only for the cloud provider.

I think other people also struggle with this issues, i.e. #48792

More precisely: Could we have a proxy-setting for the cloud provider level, i.e. in cloud-provider.ini? Is this something that can be implemented autonomously on the cloud provider level, i.e. the VSphere guys could add such a functionality themselves?

[MatthiasWinzeler]

/sig area/cloudprovider
/sig area/platform/vsphere

[dims]

/area cloudprovider
/area platform/vsphere
/sig cluster-lifecycle

[djsly]

/area platform/azure

We have the same requirements in Azure and currently using the HTTP_PROXY,HTTPS_PROXY and NO_PROXY solution doesn't feel right...

In our case (azure specific) we would like that we could control the access using the Azure Network Security Group but this is another request. Until Microsoft provides us a better way to open up the access to the cloud API endpoints, we have to use the HTTP_PROXY approach and the nightmare of managing the NO_PROXY entries.

[shinji62]

/area platform/openstack

Same issue with Openstack, I am willing to add the support in Openstack Cloud provider does this seems to be acceptable ?

[khenidak]

@djsly while i understand the need for proxy in an on-prem deployment. I really don't see a use case for it on the cloud. Can you please expand more? is this for masters + nodes to ARM?

[djsly]

@khenidak thanks for your reply, may you elaborate on the reasons why you do not see the use case with the cloud? Currently our security requirement enforces us to setup an outbound rule on all of our NSGs (nodes/master) to prevent the internet access. (e.g. DENY outbound internet)

Since both kubelet/controller-manager are accessing the ARM api and the LOGIN api to perform multiple interaction with the cloud provider, we would need a special service TAG at the NSG level to allow all connection to Azure Management endpoints.

[khenidak]

Thanks @djsly for the context. are you currently solving for this by running a proxy where all kubelet/controller->proxy->ARM/Login endpoint?

[djsly]

@khenidak, that's correct. We updated our kubelet and kube-controller-manager systemd environment file to use an external squid http proxy.

{% set http_proxy = salt.pillar.get('kubernetes:kube-system:http_proxy','') -%}
{% if http_proxy != '' -%}
HTTP_PROXY='{{ http_proxy }}'
HTTPS_PROXY='{{ http_proxy }}'
{% set no_proxy = salt.pillar.get('kubernetes:kube-system:no_proxy')|join(",")~",."~salt.cmd.run('dnsdomainname') -%}
NO_PROXY='{{ no_proxy }}'
{% endif -%}

It would make more sense if we could leverage a NSG service tag for all of Azure internal endpoints.
(when we know that the traffic isn't / shouldn't be leaving the internal network)

I guess this solution would fall outside of this particular issue #56569 and be a Azure specific solution.

[khenidak]

Yes - I think the cloud itself should take care of this (maybe via Service Endpoint, or similar), I will push the feedback internally

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close