Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Where does kubeadm take the proxy settings from? #324

Closed
erikbgithub opened this issue Jun 28, 2017 · 13 comments

Comments

Projects
None yet
10 participants
@erikbgithub
Copy link

commented Jun 28, 2017

It's not /etc/environment, it's not the current bash session that kubeadm is running in, it's not docker or kubelet environment. I verified this by setting no_proxy to a different value in all these instances. And for some reason after a kubeadm init it still continues to set another value for no_proxy. Restart, daemon-reload, restarting the services all doesn't change that fact.

Honestly it's really annoying that it only prints the line "the ip address fo.oo.ba.rr has a proxy set to blubb" instead of saying where it takes the value from. And why doesn't it simply read the value from /etc/environment, which is the one true source of truth when it comes to proxy setting, or the current bash session in which I call kubeadm which is the easiest place to make changes to?

What I expect would be something like this:

  1. kubeadm checks the current env variable http_proxy. (or https_proxy if secure communication is configured)
  2. kubeadm checks the current env variable HTTP_PROXY and warns if it is different.
  3. kubeadm checks http_proxy in /etc/environment. It warns if it is different.
  4. similar for upper case.
  5. if there is neither variable in neither context it assumes there is no proxy and informs about it.
  6. kubeadm writes the manifest files (I assume this is done before creating the docker containers) with the given proxy, giving preference to the process environment setting in lower case, then upper case, then lower case /etc/environment, then upper case /etc/environment.
  7. kubeadm starts the pods.
  8. kubeadm checks whether the controller-manager can talk to the api server. If it gets a "forbidden" or "timeout" it assumes the proxy settings are wrong and erros, calling a kubeadm reset internally.
  9. there is no event where it just waits forever without any output. It can at least reasonably well figure out if there are continuous errors in the logs of api-server and controller-manager, as well as whether there are any new logs for >= 10 minutes. And then it can error out with a corresponding error message.
  10. kubeadm internally prepends the advertise address to all no_proxy settings (add the end it may get cut of). <-- Also it would be so much better to use a hostname if possible, since no_proxy is actually meant for names, not IPs.

I seriously can't express how many working hours it would save people in enterprise networks.

@luxas

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

@erikbgithub Thanks a lot for this issue!
Up front I must say that I'm no proxy expert as I haven't experimented in such environments much.

So I can't comment on the exact statements above really, but I'd be very glad if you wanted to contribute to kubeadm to make the behavior behind a proxy better.

To answer your question, here is the relevant go code:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L432

func getProxyEnvVars() []v1.EnvVar {
	envs := []v1.EnvVar{}
	for _, env := range os.Environ() {
		pos := strings.Index(env, "=")
		if pos == -1 {
			// malformed environment variable, skip it.
			continue
		}
		name := env[:pos]
		value := env[pos+1:]
		if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" {
			envVar := v1.EnvVar{Name: name, Value: value}
			envs = append(envs, envVar)
		}
	}
	return envs
}

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/preflight/checks.go#L291

// HTTPProxyCheck checks if https connection to specific host is going
// to be done directly or over proxy. If proxy detected, it will return warning.
type HTTPProxyCheck struct {
	Proto string
	Host  string
	Port  int
}

func (hst HTTPProxyCheck) Check() (warnings, errors []error) {

	url := fmt.Sprintf("%s://%s:%d", hst.Proto, hst.Host, hst.Port)

	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return nil, []error{err}
	}

	proxy, err := http.DefaultTransport.(*http.Transport).Proxy(req)
	if err != nil {
		return nil, []error{err}
	}
	if proxy != nil {
		return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
	}
	return nil, nil
}

I seriously can't express how many working hours it would save people in enterprise networks.

Couldn't agree more

@luxas

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

@erikbgithub

This comment has been minimized.

Copy link
Author

commented Jul 5, 2017

@luxas Thanks I'll work through that when I get a round tuit. Before I can supply patches I need to learn some go though, so I would appreciate if others can churn in for now. ;-)

First sub-question I'll look into is what go actually gets via os.Environ().

@luxas

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

@erikbgithub Let me know if you need some help with creating patches and I'll help

@kad

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

@erikbgithub as original author of that check, I'll be happy to answer any questions.
First few answers:

  • kubeadm gets and checks environment from your currently running session. You can see what do you have if you execute $ env | grep -i _proxy= | sort. E.g. inside our company firewall I have something like this:
$ env | grep -i _proxy= | sort
ALL_PROXY=http://proxy-ir.example.com:911
FTP_PROXY=http://proxy-ir.example.com:911
HTTPS_PROXY=http://proxy-ir.example.com:911
HTTP_PROXY=http://proxy-ir.example.com:911
NO_PROXY=.example.com
all_proxy=http://proxy-ir.example.com:911
ftp_proxy=http://proxy-ir.example.com:911
http_proxy=http://proxy-ir.example.com:911
https_proxy=http://proxy-ir.example.com:911
no_proxy=.example.com
$
  • Usual issue that people are stepping on without realizing it, that no_proxy variable does NOT support network ranges. so putting something like NO_PROXY=10.0.0.0/8, 192.168.0.0/16 will not have any effect and will still produce warning in pre-flight check.
  • Content difference between upper/lowercase *_proxy variables doesn't matter. Go code that handles proxy environment variables have its internal logic in which order it processes it (as I recall, first upper case, then lower case, but that shouldn't matter, you can't guarantee in which order each app processing them)
  • files like /etc/environment are distro-specific and not read by each individual binary. they are read and injected into process environment variables by login scripts, PAM modules, etc. Environment variables can be also coming from e.g. SSH sessions or from external management tools (like ansible) while calling other binaries, like kubeadm. So, be dependant on particular /etc/environment or similar is neither feasible nor logical.
  • For 8 point, preflight check is specifically about that. It gives warning if kubeadm detects that it is going over proxy to API server. While it was discussion about this preflight check initially, it was agreed that those cases might be legitimate, thus it only warns, not producing error.
  • for 10: it was also objections about manipulating those variables from multiple people. (my initial idea was at all to drop all *_proxy settings to force direct connections, but it was turned down, as it might be legitimate reasons to connect over proxies).
@ErikOShaughnessy

This comment has been minimized.

Copy link

commented Aug 23, 2017

I "fixed" this problem by including all my cluster node IPs in NO_PROXY and using the same NO_PROXY on all the minions when joining the cluster.

$ export NO_PROXY='ip,ip,ip,ip,.example.com'
[master]$ kubeadm init
[minion]$ kubeadm join --token={token} a.b.c.d:6443

To be honest, I'm not sure if it's all the IP addresses being enumerated or the .example.com that fixed the problem.

@kad

This comment has been minimized.

Copy link
Member

commented Sep 22, 2017

if PR kubernetes/kubernetes#52788 will be merged, it will be possible to specify in NO_PROXY IP ranges for your nodes. it will simplify things a lot.

@ted-jung

This comment has been minimized.

Copy link

commented Dec 5, 2017

A little bit weired. if i look into the code "checks.go".
it always return error message if there are value in proxy.

if proxy != nil {
return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
}
return nil, nil

In enterprise...there are necessarily three proxy options. (http_proxy, https_proxy, no_proxy)
http_* is mandatory option to pull images for connection to internet.
if there is no_proxy option be set...then it should return error message.

" pl, set option(no_proxy) not to be routed to the proxy for internal connection"

@rmxhaha

This comment has been minimized.

Copy link

commented Jan 9, 2018

I want to ask if kubeadm join supports http_proxy ?

I manage to get kubeadm init to work with http_proxy and no_proxy but it seems kubeadm join produce errors such as

kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Get https://gcr.io/v1/_ping: read tcp <my-ip>:58742->74.125.68.82:443: read: connection reset by peer

and also
/etc/environment is empty instead of filled with configuration like in the master.

which let me to believe maybe http_proxy and no_proxy is not yet supported for kubeadm join.

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Jan 20, 2018

Kubernetes Submit Queue
Merge pull request #53895 from kad/kubeadm-proxy-transports
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

kubeadm: Utilize transport defaults from API machinery for http calls inside kubeadm

**What this PR does / why we need it**:
Default Go HTTP transport does not allow to use CIDR notations in
NO_PROXY variables, thus for certain HTTP calls that is done inside
kubeadm user needs to put explicitly multiple IP addresses. For most of
calls done via API machinery it is get solved by setting different Proxy
resolver. This patch allows to use CIDR notations in NO_PROXY variables
for currently all other HTTP calls that is made inside kubeadm.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes kubernetes/kubeadm#324

**Special notes for your reviewer**:
Based on discussion in #52788, replacing this patch replacing all calls inside kubeadm that are done via DefaultTransport to explicitly defined and initialized with API machinery defaults Transport and http client.

**Release note**:
```release-note
- kubeadm now supports CIDR notations in NO_PROXY environment variable
```
@erikbsap

This comment has been minimized.

Copy link

commented May 9, 2018

Running into this problem once more. It still uses the proxy incorrectly and I seem to be unable to modify proxy and no_proxy settings.

@PhilippeDo

This comment has been minimized.

Copy link

commented Apr 8, 2019

From my experience, kubeadm use the proxy defined in /etc/environment

@earthling42

This comment has been minimized.

Copy link

commented Apr 11, 2019

From my experience, kubeadm use the proxy defined in /etc/environment

Yup - in my case it is also /etc/environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.