New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container on Windows agent doesn't have outbound internet connection #959

Closed
robbiezhang opened this Issue Jul 10, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@robbiezhang
Copy link

robbiezhang commented Jul 10, 2017

Is this a request for help?:
Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
Issue

What version of acs-engine?:
v0.3.0

The container on Windows agent doesn't have outbound internet connection. Bascially, it cannot resolve the DNS.

PS C:\Agent> Invoke-WebRequest -Uri http://bing.com -UseBasicParsing
Invoke-WebRequest : The remote name could not be resolved: 'bing.com'
At line:1 char:1
+ Invoke-WebRequest -Uri http://bing.com -UseBasicParsing
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

However, if I RDP to the machine, and use the docker to create the same image, it can resolve the DNS and get the result.

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.6.6

What happened:
No outbound internet connection from container.

What you expected to happen:
Outbound internet connections should be allow in the container.

How to reproduce it (as minimally and precisely as possible):
Deploy a cluster with Windows agent.

Anything else we need to know:

@seanknox

This comment has been minimized.

Copy link
Member

seanknox commented Jul 11, 2017

Hey @robbiezhang, which version were you using prior to v0.3.0? I don't see anything suspicious from v0.2.0, so I suspect something snuck in prior to that.

@robbiezhang

This comment has been minimized.

Copy link

robbiezhang commented Jul 11, 2017

Talked to @JiangtianLi, it seems like an issue in k8s 1.6.6. Previously, we use ACS-Engine v0.1.2 with k8s 1.6.2 and 1.5.3, it works fine.

@seanknox

This comment has been minimized.

Copy link
Member

seanknox commented Jul 11, 2017

Hmm, that'd be a nasty upstream regression. Try building a cluster using acs-engine v0.3.0 and k8s v1.6.2 and see if this issue goes away.

@robbiezhang

This comment has been minimized.

Copy link

robbiezhang commented Jul 11, 2017

I can try this combination. However, I need CloudProvider back off change, which is only available on v1.6.6+.

@seanknox

This comment has been minimized.

Copy link
Member

seanknox commented Jul 11, 2017

Understood. Cloud provider support is a known quantity, so let's try to isolate where outbound internet broke with Windows.

@yangl900

This comment has been minimized.

Copy link

yangl900 commented Jul 11, 2017

btw - I noticed although master node running 1.6.6, the kubelet version on windows node is not:

Kubelet Version: v1.6.4-beta.0.147+2b3fc68910b4dd

@JiangtianLi @seanknox is this expected? why we run a v1.6.4 kubelet on windows node? I have specified orchestrator version 1.6.6 in apimodel.

@seanknox

This comment has been minimized.

Copy link
Member

seanknox commented Jul 11, 2017

That would be a bug--nodes and masters should always run the same version. Cc @Azure/acs-engine-core

@JiangtianLi

This comment has been minimized.

Copy link
Contributor

JiangtianLi commented Jul 11, 2017

@yangl900 I built the kublet binary out of https://github.com/JiangtianLi/kubernetes/tree/k8swin166 and I did notice that the kubelet version didn't show as 1.6.6 but 1.6.4 beta at that time. I discussed with Anthony about it. Can you build out of my branch and see what kubelet.exe --version say? Also I just built again and due to the Go version is bumped from 1.7.3 (https://github.com/Azure/acs-engine/blob/master/docs/kubernetes-build-win-binaries.md) to 1.8.3 after I built v1.7.0 recently, I found the kubelet version now show as 1.6.6. So if you have go version 1.7.3, please build with that first and verify.

@robbiezhang

This comment has been minimized.

Copy link

robbiezhang commented Jul 11, 2017

Repo the same issue in acs-engine v0.3.0 + k8s v1.6.2
The Windows hyperKube version is v1.6.0-alpha.1.2959+451473d43a2072
@seanknox @JiangtianLi

@seanknox

This comment has been minimized.

Copy link
Member

seanknox commented Jul 11, 2017

Repo the same issue in acs-engine v0.3.0 + k8s v1.6.2
The Windows hyperKube version is v1.6.0-alpha.1.2959+451473d43a2072

@robbiezhang typo^^ possibly? The Windows version is v1.6.0 though you referred to v1.6.2 above.

@JackQuincy

This comment has been minimized.

Copy link
Member

JackQuincy commented Jul 11, 2017

Repo the same issue in acs-engine v0.3.0 + k8s v1.6.2 The Windows hyperKube version is v1.6.0-alpha.1.2959+451473d43a2072

The differences is a known issue with how we generate our hypercube images. We don't properly set the version on them.

@robbiezhang

This comment has been minimized.

Copy link

robbiezhang commented Jul 11, 2017

@seanknox not a typo. I use 1.6.2 in the API model.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment