Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

switched to use the pause image served from gcr #84538

Merged
merged 2 commits into from Oct 31, 2019
Merged

switched to use the pause image served from gcr #84538

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f574071
switched to use the pause image served from gcr
yliaog Oct 29, 2019
6ca8bd4
moved Pull-InfraContainer to the last
yliaog Oct 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion cluster/gce/windows/configure.ps1
View file
Open in desktop
Expand Up @@ -140,7 +140,6 @@ try {

Create-DockerRegistryKey
Configure-Dockerd
Pull-InfraContainer
DownloadAndInstall-KubernetesBinaries
Create-NodePki
Create-KubeletKubeconfig
Expand All @@ -159,6 +158,8 @@ try {

$config = New-FileRotationConfig
Schedule-LogRotation -Pattern '.*\.log$' -Path ${env:LOGS_DIR} -RepetitionInterval $(New-Timespan -Hour 1) -Config $config

Pull-InfraContainer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this change?

On one hand I suppose this should let the node join the cluster more quickly. On the other hand there could be some delay if a user tries to run a Windows pod immediately after the node is Ready but the pause container hasn't finished pulling yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is not needed for kubelet/node to register with the master/control plane. Pull-InfraContainer is a best-effort optimization, so that when a user start a pod, the 'pause' image is likely already present on the node. Moving it to the last would 1) shorten the delay for the node to join the cluster, [so user can submit deployment/pod earlier, user does not need to wait for the node to be ready to submit deployment/pod in theory, but in practice, user may wait/check for node to be ready before deploying anything] 2) no worse than current situation (should be better) in terms of user-perceived node startup time. [we may explore moving it to the first step when making it async/pulling in the background]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks for clarifying.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem safe to move this.

Windows image pull is slow, for docker pulling the pause image can takes >4min, for containerd it is even longer...

If the pause image is not prepulled before the node is ready, it is possible that a user will never be able to create a pod.

The reason is that in kubelet we set a 4min timeout for pod sandbox start (including pulling the pause image) https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/remote/remote_runtime.go#L97.

If the pause image is not prepulled, RunPodSandbox may timeout every time, abort the ongoing pulling, and try pulling again. If for any reason the pause image pull is longer than 4min, the image pull may be always aborted, thus the pod can't start any more.

For Docker, I believe this may happen, and for containerd, because of the slow image pull this breaks the test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the pause image is not prepulled, RunPodSandbox may timeout every time, abort the ongoing pulling, and try pulling again.

This doesn't sound like the right thing to do on Windows. Can/should we change that behavior?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chatted offline about it.

}
catch {
Write-Host 'Exception caught in script:'
Expand Down
2 changes: 1 addition & 1 deletion cluster/gce/windows/k8s-node-setup.psm1
View file
Open in desktop
Expand Up @@ -51,7 +51,7 @@
# - Document functions using proper syntax:
# https://technet.microsoft.com/en-us/library/hh847834(v=wps.620).aspx

$INFRA_CONTAINER = 'mcr.microsoft.com/k8s/core/pause:1.0.0'
$INFRA_CONTAINER = 'gcr.io/gke-release/pause-win:1.0.0'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify: is this GCR bucket available to any user, not just google-owned projects?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is available to public.

$GCE_METADATA_SERVER = "169.254.169.254"
# The "management" interface is used by the kubelet and by Windows pods to talk
# to the rest of the Kubernetes cluster *without NAT*. This interface does not
Expand Down