Cluster cannot start with older versions of k8s #10122

hakman · 2020-10-27T17:01:32Z

After merging #10048, the new way of installing Docker is conflicting with the pre-installed Docker in the kope.io images.
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/e2e-kops-aws-k8s-1-15/1320685447990480896

A similar problem existed in 1.17 and was fixed in kubernetes-sigs/image-builder#273, by just copying the Docker deb files to /var/cache/nodeup/packages and letting kOps install them if the hash matches.

All kope.io images from 1.11 to 1.16, maybe also 1.17.

The text was updated successfully, but these errors were encountered:

rifelpet · 2020-12-31T14:08:14Z

This issue is impacting kubernetes 1.17 as well: https://testgrid.k8s.io/kops-versions#kops-aws-k8s-1.17

https://storage.googleapis.com/kubernetes-jenkins/logs/e2e-kops-aws-k8s-1-17/1344395270774853632/artifacts/18.193.88.194/protokube.log

Starting Kubernetes Protokube Service...
Started Kubernetes Protokube Service.
ctr: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/run/dbus\\\" to rootfs \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/protokube/rootfs\\\" at \\\"/var/run/dbus\\\" caused \\\"stat /var/run/dbus: no such file or directory\\\"\"": unknown
protokube.service: Main process exited, code=exited, status=1/FAILURE

hakman · 2020-12-31T14:15:57Z

Yes, dbus should be installed on those images.
Ref: kubernetes-sigs/image-builder#130

By creating a tempfile and then moving the file into place, we both write more atomically and we can overwrite in-use files. Issue kubernetes#10122

Issue kubernetes#10122

atam003 · 2021-01-26T04:54:05Z

I am using kops1.19 +Kubernetes 1.16. I tried using image latest available kops image "kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-11-19". I still see this issue.
Is there any latest published image of kops which can be used?

hakman · 2021-01-26T06:08:13Z

@atam003 until 1.19 is released, you can use the official Debian 9 image or the Ubuntu 20.04 image.

sonowz · 2021-01-28T05:15:27Z

Is there any workaround with this issue?
I'm using kops 1.19-beta3, Kubernetes 1.18, and my AMI is Ubuntu 18.04 with Docker pre-installed.
Then struck into same error as #10492.
What should I do in order for the AMI to work? Does upgrading Kubernetes into 1.20 solves the issue?

hakman · 2021-01-28T05:45:08Z

@sonowz can you explain a bit more why Ubuntu 18.04 and why docker pre-installed?

As for workaround, you could try with:

  containerd:
    skipInstall: true
  docker:
    skipInstall: true

The issue should be fixed once 1.19 is released later this week (I hope).

sonowz · 2021-01-28T06:13:45Z

There's no reason to use 18.04, it's just a version I'm currently using.
The reasons why Docker is pre-installed are the following:

nvidia-container-runtime should be configured as default runtime
Some images should be stored in AMI due to its gigantic size (~20GB)

The issue should be fixed once 1.19 is released later this week (I hope).

That's good to hear! I'm looking forward to it.

hakman · 2021-01-28T06:26:16Z

Thanks for explaining. You may also disable the Docker and containerd services in the AMI. kOps will start them later.

sonowz · 2021-01-28T10:46:48Z

Disabling autostart of Docker and containerd service works! Thanks a lot 👍

atam003 · 2021-01-29T22:57:05Z

@hakman
I see kops 1.19 is released today. I tried using it. I see the issue is still there ... Please note that I am using Kubernetes 1.16.9 version and tried the image version - kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-07-20

By creating a tempfile and then moving the file into place, we both write more atomically and we can overwrite in-use files. Issue kubernetes#10122

hakman · 2021-01-30T09:37:13Z

Thanks @atam003, #10691 should fix this in 1.19 also.

atam003 · 2021-01-30T19:44:18Z

@hakman I still see the same issue ...

Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.909736 846 changes.go:81] Field changed "Mode" actual="0777" expected="0755"
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.909751 846 changes.go:81] Field changed "Type" actual="symlink" expected="file"
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.909821 846 files.go:50] Writing file "/usr/bin/dockerd"
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.953492 846 changes.go:81] Field changed "Contents" actual="\u007fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00>\x00\x01\x00\x00\x00\xf0\xc7\xe4\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00U\xec\x02\x00\x00\x00\x00\x00\x00\x00\x00@\x008\x00\n\x00@\x00\x1f\x00\x1e\x00\x06\x00\x00\x00\x05\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x000\x02\x00\x00\x00\x00\x00\x000\x02\x00\x00\x00\x00\x00\x00\b\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00p\x02\x00\x00\x00\x00\x00\x00p\x02\x00\x00\x00\x00\x00\x00p\x02\x00\x00\x00\x00\x00\x00\x1c\x00\x00\x00\x00\x00\x00\x00\x1c\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc/\xe0\x01\x00\x00\x00\x00\xbc/\xe0\x01\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x01\x00\x00\x00\x06\x00\x00\x00\xa8;\xe0\x01\x00\x00\x00\x00\xa8;\x00\x02\x00\x00\x00\x00... (truncated)" expected="\u007fELF\x02\x01\x01\x03\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00>\x00\x01\x00\x00\x00\x10\n@\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x18\x11\xeb\x01\x00\x00\x00\x00\x00\x00\x00\x00@\x008\x00\x06\x00@\x00"\x00!\x00\x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x12\xf6\x00\x00\x00\x00\x00\x00\x12\xf6\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x01\x00\x00\x00\x06\x00\x00\x00\xe8\x18\xf6\x00\x00\x00\x00\x00\xe8\x18V\x01\x00\x00\x00\x00\xe8\x18V\x01\x00\x00\x00\x00y\xf6\xf4\x00\x00\x00\x00\x00\x80\xd0\xf7\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x90\x01\x00\x00\x00\x00\x00\x00\x90\x01@\x00\x00\x00\x00\x00\x90\x01@\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\a\x00\x00\x00\x04\x00\x00\x00\xe8\x18\xf6\x00\x00\x00\x00\x00\xe8\x18V\x01\x00\x00\x00\x00... (truncated)"
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.953661 846 files.go:50] Writing file "/usr/bin/containerd"
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: W0130 19:43:11.954079 846 executor.go:139] error running task "File//usr/bin/dockerd" (7m4s remaining to succeed): error copying file "/usr/bin/dockerd": error opening destination file "/usr/bin/dockerd": open /usr/bin/dockerd: text file busy
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: W0130 19:43:11.954111 846 executor.go:139] error running task "File//usr/bin/containerd" (7m4s remaining to succeed): error copying file "/usr/bin/containerd": error opening destination file "/usr/bin/containerd": open /usr/bin/containerd: text file busy
Jan 30 19:43:11 ip-172-20-37-61 nodeup[846]: I0130 19:43:11.954121 846 executor.go:155] No progress made, sleeping before retrying 2 task(s)

atam003 · 2021-01-30T22:59:32Z

@hakman Looks changes are not picked. Particular libs (Nodeup) are picked from repositories.
I am building my kops using the following doc.
https://kops.sigs.k8s.io/contributing/building/

Can you please help with that?

atam003 · 2021-02-03T07:56:15Z

@hakman @justinsb Any update on it. Looks like Nodeup has to be updated in repository with these changes?

To pick up the sudo fixes, also addresses some problems with docker preinstallation. Issue kubernetes#10122

fejta-bot · 2021-07-07T08:13:39Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

k8s-triage-robot · 2021-08-06T08:16:17Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

k8s-triage-robot · 2021-09-27T01:06:21Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-09-27T01:06:36Z

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hakman assigned justinsb Oct 27, 2020

hakman added this to the v1.18 milestone Oct 27, 2020

hakman added blocks-next kind/bug Categorizes issue or PR as related to a bug. labels Oct 27, 2020

hakman modified the milestones: v1.18, v1.19 Oct 27, 2020

hakman changed the title ~~cluster cannot start on older versions of k8s~~ Cluster cannot start on older versions of k8s Oct 27, 2020

hakman changed the title ~~Cluster cannot start on older versions of k8s~~ Cluster cannot start with older versions of k8s Oct 27, 2020

johngmyers mentioned this issue Oct 28, 2020

Automated cherry pick of #10048: Install container runtime packages as assets #10123

Merged

hakman closed this as completed Dec 4, 2020

hakman reopened this Dec 4, 2020

hakman mentioned this issue Dec 22, 2020

nodeup failing to update containerd and dockerd files (and node wont start) #10492

Closed

justinsb mentioned this issue Jan 14, 2021

Allow nodeup (and others) to replace in-use files #10581

Merged

justinsb added a commit to justinsb/kops that referenced this issue Jan 14, 2021

Allow nodeup (and others) to replace in-use files

acb247f

By creating a tempfile and then moving the file into place, we both write more atomically and we can overwrite in-use files. Issue kubernetes#10122

justinsb added a commit to justinsb/kops that referenced this issue Jan 14, 2021

Install dbus if needed for protokube with containerd

69c39ed

Issue kubernetes#10122

justinsb mentioned this issue Jan 14, 2021

Install dbus if needed for protokube with containerd #10583

Merged

hakman pushed a commit to hakman/kops that referenced this issue Jan 30, 2021

Allow nodeup (and others) to replace in-use files

2c2fd3f

By creating a tempfile and then moving the file into place, we both write more atomically and we can overwrite in-use files. Issue kubernetes#10122

justinsb mentioned this issue Feb 6, 2021

alpha channel: Update older images #10748

Merged

justinsb added a commit to justinsb/kops that referenced this issue Feb 6, 2021

alpha channel: Update older images

fce17c4

To pick up the sudo fixes, also addresses some problems with docker preinstallation. Issue kubernetes#10122

hakman removed the blocks-next label Mar 16, 2021

olemarkus modified the milestones: v1.19, v1.22 Apr 8, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 7, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 6, 2021

johngmyers removed this from the v1.22 milestone Aug 28, 2021

k8s-ci-robot closed this as completed Sep 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster cannot start with older versions of k8s #10122

Cluster cannot start with older versions of k8s #10122

hakman commented Oct 27, 2020

rifelpet commented Dec 31, 2020

hakman commented Dec 31, 2020

atam003 commented Jan 26, 2021 •

edited

Loading

hakman commented Jan 26, 2021

sonowz commented Jan 28, 2021 •

edited

Loading

hakman commented Jan 28, 2021

sonowz commented Jan 28, 2021

hakman commented Jan 28, 2021

sonowz commented Jan 28, 2021

atam003 commented Jan 29, 2021 •

edited

Loading

hakman commented Jan 30, 2021

atam003 commented Jan 30, 2021

atam003 commented Jan 30, 2021 •

edited

Loading

atam003 commented Feb 3, 2021

fejta-bot commented Jul 7, 2021

k8s-triage-robot commented Aug 6, 2021

k8s-triage-robot commented Sep 27, 2021

k8s-ci-robot commented Sep 27, 2021

Cluster cannot start with older versions of k8s #10122

Cluster cannot start with older versions of k8s #10122

Comments

hakman commented Oct 27, 2020

rifelpet commented Dec 31, 2020

hakman commented Dec 31, 2020

atam003 commented Jan 26, 2021 • edited Loading

hakman commented Jan 26, 2021

sonowz commented Jan 28, 2021 • edited Loading

hakman commented Jan 28, 2021

sonowz commented Jan 28, 2021

hakman commented Jan 28, 2021

sonowz commented Jan 28, 2021

atam003 commented Jan 29, 2021 • edited Loading

hakman commented Jan 30, 2021

atam003 commented Jan 30, 2021

atam003 commented Jan 30, 2021 • edited Loading

atam003 commented Feb 3, 2021

fejta-bot commented Jul 7, 2021

k8s-triage-robot commented Aug 6, 2021

k8s-triage-robot commented Sep 27, 2021

k8s-ci-robot commented Sep 27, 2021

atam003 commented Jan 26, 2021 •

edited

Loading

sonowz commented Jan 28, 2021 •

edited

Loading

atam003 commented Jan 29, 2021 •

edited

Loading

atam003 commented Jan 30, 2021 •

edited

Loading