Switch windows runtime endpoints to npipe #69516

feiskyer · 2018-10-08T09:20:20Z

What this PR does / why we need it:

Switch default windows runtime endpoints to npipe. tcp endpoints are also supported for backward compatibility.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Refer kubernetes-sigs/cri-tools#374.

Release note:

Windows runtime endpoints is now switched to 'npipe:////./pipe/dockershim' from 'tcp://localhost:3735'.

/sig windows
/assign @PatrickLang

feiskyer · 2018-10-08T11:41:01Z

cc @jterry75

/retest

jterry75 · 2018-10-09T17:07:18Z

@feiskyer - Can you take a look at: jterry75@a2f1f44. I did npipe support a while back including adding tests and supporting both path types (npipe://./path and \\.\pipe\path). I think that we should merge your commit with this one so we get all of it. Do you agree?

feiskyer · 2018-10-10T01:53:07Z

npipe://./path and \.\pipe\path

I'd suggest we only support npipe format. The reason it this is an option of kubelet, so it's format should be consistent with all platforms, e.g. unix:... for linux and npipe:... for windows.

feiskyer · 2018-10-10T01:53:16Z

/retest

Random-Liu · 2018-10-10T06:45:56Z

@jterry75 I do think npipe:... is more clear to me. Is there any particular reason that \\.\pipe\path should be supported? For example, compatibility? Or in some case only that works?

feiskyer · 2018-10-10T09:02:43Z

Is there any particular reason that \.\pipe\path should be supported? For example, compatibility?

I think backward compatibility only exists on volumes, e.g. what #69484 did. Kubelet options are new features, there are no backward issues.

jterry75 · 2018-10-10T17:19:19Z

@Random-Liu @feiskyer - I am totally fine supporting only npipe:// paths. But we still need this part of the code: jterry75@a2f1f44#diff-ae2806b52080b15d453c30c075eb0bc8R47. And I would prefer you take the test cases that I added for npipe:// paths.

IE:

protocol, addr, err := parseEndpoint("npipe://./pipe/mypipe")
// protocol == "npipe"
// addr == "\\\\.\\pipe\\mypipe"

We need to return addr as the \\.\pipe converted path so that winio.DialPipe or winio.ListenPipe works.

feiskyer · 2018-10-11T04:24:48Z

@jterry75 Thanks. But the commit you provided doesn't handle npipe:////./pipe/ correctly. I fixed that another way. PTAL

jterry75 · 2018-10-11T04:51:47Z

@feiskyer - Your change looks good. Two questions:

Is npipe:////./pipe (4 /'s) even a valid addr? I understand that npipe://./pipe (2 /'s) is valid and should be parse-able though. That would be like putting tcp:////127.0.0.1:8080 right?
Are you saying that winio.DialPipe("//./pipe/mypipe", nil) works? I haven't tried it with forward slashes but I thought Windows paths had to be winio.DialPipe("\\\\.\\pipe\\mypipe", nil) which is why I suggested returning the addr portion from parse normalized.

feiskyer · 2018-10-11T04:58:47Z

For your question:

npipe:////./pipe is valid only for npipe and the change doesn't influence tcp
winio.DialPipe("//./pipe/mypipe", nil) works of course for windows. It is actually same with winio.DialPipe("\\\\.\\pipe\\mypipe", nil)

jterry75 · 2018-10-11T13:44:54Z

Perfect. LGTM then

feiskyer · 2018-10-12T06:20:21Z

@Random-Liu PTAL

btw, cri-tools should be updated after this PR merged. I will update it after this PR got merged.

yujuhong · 2018-10-15T20:03:29Z

pkg/kubelet/util/util.go

@@ -30,6 +31,8 @@ func FromApiserverCache(opts *metav1.GetOptions) {
 }

 func parseEndpoint(endpoint string) (string, string, error) {
+	// url.Parse doesn't recognize \, so replace with / first.
+	endpoint = strings.Replace(endpoint, "\\", "/", -1)


I don't think it's safe to do this on non-windows platforms.

Either use libraries that can handle paths differently on different OS, or rewrite this function in _windows.go
The latter might be good since protocols like unix:// cannot be accepted on windows.

Good catch. Thanks.

yujuhong · 2018-10-15T20:05:26Z

/cc @pjh

k8s-ci-robot · 2018-10-15T20:05:27Z

@yujuhong: GitHub didn't allow me to request PR reviews from the following users: pjh.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @pjh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

feiskyer · 2018-10-16T04:57:24Z

@yujuhong Addressed comments. PTAL

yujuhong

Okay with the overall approach. Would like a windows folk to review the npipe path parsing

yujuhong · 2018-10-18T17:52:28Z

pkg/kubelet/util/util_unix.go

+		return "", "", err
+	}
+
+	if u.Scheme == "tcp" {


Hmm... we can just use switch here.

yujuhong · 2018-10-18T17:53:38Z

cmd/kubelet/app/options/options.go

@@ -376,8 +376,8 @@ func (f *KubeletFlags) AddFlags(mainfs *pflag.FlagSet) {
 	fs.StringVar(&f.ExperimentalMounterPath, "experimental-mounter-path", f.ExperimentalMounterPath, "[Experimental] Path of mounter binary. Leave empty to use the default mount.")
 	fs.StringSliceVar(&f.AllowedUnsafeSysctls, "allowed-unsafe-sysctls", f.AllowedUnsafeSysctls, "Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. Sysctls feature gate is enabled by default.")
 	fs.BoolVar(&f.ExperimentalKernelMemcgNotification, "experimental-kernel-memcg-notification", f.ExperimentalKernelMemcgNotification, "If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling.")
-	fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'")
-	fs.StringVar(&f.RemoteImageEndpoint, "image-service-endpoint", f.RemoteImageEndpoint, "[Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'")
+	fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'")


npipe is also supported on windows, no?

Also, I guess tcp is supported on both platforms?

Good catch. The usage should include such information.

yujuhong · 2018-10-18T17:53:58Z

cmd/kubelet/app/options/options.go

-	fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'")
-	fs.StringVar(&f.RemoteImageEndpoint, "image-service-endpoint", f.RemoteImageEndpoint, "[Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'")
+	fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'")
+	fs.StringVar(&f.RemoteImageEndpoint, "image-service-endpoint", f.RemoteImageEndpoint, "[Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'")


yujuhong · 2018-10-18T17:59:47Z

pkg/kubelet/util/util_windows_test.go

+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+


Do we have pre-submit tests running windows unit tests?

Not yet. @PatrickLang is driving Node e2e tests for Windows, but it's still running outside. We hope this will be done within v1.13 release cycle.

yujuhong · 2018-10-18T18:00:47Z

pkg/kubelet/util/util_windows_test.go

+			assert.NotNil(t, err, "Expect error during parsing %q", test.endpoint)
+			continue
+		}
+		assert.Nil(t, err, "Expect no error during parsing %q", test.endpoint)


require.Nil is better here since you don't want the test to continue if parsing failed.

yujuhong · 2018-10-18T18:03:36Z

pkg/kubelet/util/util_windows_test.go

+			expectedAddr:     "//./pipe/mypipe",
+		},
+		{
+			endpoint:         "npipe:////./pipe/mypipe2",


Curious, since I don't know much about the npipe and windows path syntax.
Is ./pipe required and what purpose does it serve?
Also, what does the // prefix do here?

Yep, pipe is required in the pipe path ( . is actually the server name for local). The full path format is \\<serverName>\pipe\<pipeName>.

See more here.

feiskyer · 2018-10-22T13:29:29Z

@yujuhong Addressed comments. PTAL

yujuhong · 2018-10-22T17:51:20Z

/approve

k8s-ci-robot · 2018-10-22T18:52:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, yujuhong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubelet/OWNERS~~ [yujuhong]
~~pkg/kubelet/OWNERS~~ [yujuhong]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

feiskyer · 2018-10-23T01:57:17Z

/retest

feiskyer · 2018-10-23T01:57:40Z

/test pull-kubernetes-integration

PatrickLang · 2018-10-23T22:00:08Z

/lgtm

Refer kubernetes/kubernetes#69516.

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Oct 8, 2018

k8s-ci-robot assigned PatrickLang Oct 8, 2018

k8s-ci-robot requested review from resouer and sjpotter October 8, 2018 09:20

feiskyer added the kind/bug Categorizes issue or PR as related to a bug. label Oct 8, 2018

k8s-ci-robot removed the needs-kind Indicates a PR lacks a `kind/foo` label and requires one. label Oct 8, 2018

feiskyer added this to the v1.13 milestone Oct 10, 2018

feiskyer force-pushed the win-npipe branch from 48efcba to 6360350 Compare October 11, 2018 04:23

This was referenced Oct 15, 2018

Migrate Kubelet --container-runtime-endpoint to kubelet.config.k8s.io or remove the flag #61674

Closed

Enhancement: Support CRI-ContainerD for Windows #69823

Closed

yujuhong reviewed Oct 15, 2018

View reviewed changes

feiskyer force-pushed the win-npipe branch from 6360350 to 90ec404 Compare October 16, 2018 03:39

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 16, 2018

feiskyer force-pushed the win-npipe branch from 90ec404 to 95d1182 Compare October 16, 2018 03:45

yujuhong reviewed Oct 18, 2018

View reviewed changes

Switch windows runtime endpoints to npipe

053b71d

feiskyer force-pushed the win-npipe branch from 95d1182 to 053b71d Compare October 22, 2018 13:29

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 23, 2018

k8s-ci-robot merged commit c0974d7 into kubernetes:master Oct 23, 2018

feiskyer deleted the win-npipe branch October 23, 2018 23:50

feiskyer added a commit to feiskyer/cri-tools that referenced this pull request Oct 31, 2018

Switch Windows default runtime endpoints to npipe

d29300b

Refer kubernetes/kubernetes#69516.

feiskyer added a commit to feiskyer/cri-tools that referenced this pull request Oct 31, 2018

Switch Windows default runtime endpoints to npipe

e3ebc8a

Refer kubernetes/kubernetes#69516.

feiskyer mentioned this pull request Oct 31, 2018

Switch Windows default runtime endpoints to npipe kubernetes-sigs/cri-tools#398

Merged

feiskyer added a commit to feiskyer/cri-tools that referenced this pull request Nov 6, 2018

Switch Windows default runtime endpoints to npipe

74d8567

Refer kubernetes/kubernetes#69516.

Switch windows runtime endpoints to npipe #69516

Switch windows runtime endpoints to npipe #69516

Conversation

feiskyer commented Oct 8, 2018

feiskyer commented Oct 8, 2018

jterry75 commented Oct 9, 2018

feiskyer commented Oct 10, 2018

feiskyer commented Oct 10, 2018

Random-Liu commented Oct 10, 2018

feiskyer commented Oct 10, 2018

jterry75 commented Oct 10, 2018

feiskyer commented Oct 11, 2018

jterry75 commented Oct 11, 2018

feiskyer commented Oct 11, 2018

jterry75 commented Oct 11, 2018

feiskyer commented Oct 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong commented Oct 15, 2018

k8s-ci-robot commented Oct 15, 2018

feiskyer commented Oct 16, 2018

yujuhong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feiskyer commented Oct 22, 2018

yujuhong commented Oct 22, 2018

k8s-ci-robot commented Oct 22, 2018

feiskyer commented Oct 23, 2018

feiskyer commented Oct 23, 2018

PatrickLang commented Oct 23, 2018