Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v1.8 hyperkube images don't allow the kubelet to mount #52789

Closed
jamiehannaford opened this issue Sep 20, 2017 · 9 comments · Fixed by #52798
Closed

v1.8 hyperkube images don't allow the kubelet to mount #52789

jamiehannaford opened this issue Sep 20, 2017 · 9 comments · Fixed by #52798
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Milestone
v1.8

Comments

@jamiehannaford
Copy link
Contributor

/kind bug

What happened:

When a cluster is provisioned using a containerised kubelet using the following images:

gcr.io/google-containers/hyperkube-amd64:v1.8.0-beta.0
gcr.io/google-containers/hyperkube-amd64:v1.8.1-beta.0

Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries:

Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled		Successfully assigned thindiskpod to node3
  1m		1m		1	{kubelet node3}				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-5h5x6" 
  1m		1m		5	{kubelet node3}				Warning		FailedMount		MountVolume.MountDevice failed for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" : executable file not found in $PATH

This executable is mkfs.ext4. This problem can be resolved by mounting paths from the host into the rkt container, specifically /usr/sbin and /usr/lib.

Is this expected? @saad-ali mentioned in #50802, that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary.

If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this?
If it's not, I'll leave this open as a tracking issue.

What you expected to happen:

The hyperkube image to work without lots of host mounts.

How to reproduce it (as minimally and precisely as possible):

Run kubelet with rkt in one of the above listed images (see CoreOS' kubelet-wrapper).

Environment:

  • Kubernetes version (use kubectl version): v1.8.0.beta-1
  • Cloud provider or hardware configuration**: OpenStack
  • OS (e.g. from /etc/os-release):
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1520.3.0
VERSION_ID=1520.3.0
BUILD_ID=2017-09-15-2017
PRETTY_NAME="Container Linux by CoreOS 1520.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
  • Kernel (e.g. uname -a):
Linux jamie-terraform-worker-1 4.13.2-coreos #1 SMP Fri Sep 15 20:02:58 UTC 2017 x86_64 AMD Opteron 23xx (Gen 3 Class Opteron) AuthenticAMD GNU/Linux
  • Install tools: tectonic-installer
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 20, 2017
@jamiehannaford
Copy link
Contributor Author

/sig storage
/sig node

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Sep 20, 2017
@jamiehannaford
Copy link
Contributor Author

Seems the base hyperkube image was changed:

https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.1/cluster/images/hyperkube/Makefile#L24
https://github.com/kubernetes/kubernetes/blob/v1.7.5/cluster/images/hyperkube/Makefile#L36

done in 66b9ae7.

/cc @ixdy

@jamiehannaford jamiehannaford mentioned this issue Sep 20, 2017
Merged
@rphillips
Copy link
Member

rphillips commented Sep 20, 2017 •
edited
Loading

The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.

rphillips pushed a commit to rphillips/kubernetes that referenced this issue Sep 20, 2017
restore e2fsprogs in hyperkube image
872a3ff 
Fixes kubernetes#52789
Fixes kubernetes#50802
@rphillips rphillips mentioned this issue Sep 20, 2017
Merged
@ericchiang
Copy link
Contributor

cc @kubernetes/sig-node-bugs
cc @kubernetes/sig-storage-bugs
cc @kubernetes/kubernetes-release-managers

This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?

@ixdy ixdy added this to the v1.8 milestone Sep 20, 2017
@ixdy ixdy added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 20, 2017
@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Labels Complete

@jamiehannaford

Issue label settings:

  • sig/node sig/storage: Issue will be escalated to these SIGs if needed.
  • priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
  • kind/bug: Fixes a bug discovered during the current release.
Additional instructions available here The commands available for adding these labels are documented here

@saad-ali
Copy link
Member

Is this expected? @saad-ali mentioned in #50802, that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary.

Containerization of mount is not going to happen until 1.9 at the earliest. So if hyperkube is broken, it should be fixed before ship.

k8s-github-robot pushed a commit that referenced this issue Sep 20, 2017
Merge pull request #52798 from rphillips/fixes/restore_e2fsprogs_hype…
d83c8bc 
…rkube

Automatic merge from submit-queue (batch tested with PRs 52477, 52790, 52798). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..

restore e2fsprogs in hyperkube image

**What this PR does / why we need it**:
Kubernetes defaults to the ext4 filesystem if no filesystem is specified. Unformatted filesystems are not able to be mounted without these tools.

The default ext{2,3,4} tools and mkfs.* utilities should be included in the hyperkube image.

**Which issue this PR fixes**: Fixes #52789 #50802

**Special notes for your reviewer**:

**Release note**:
```release-note
NONE
```
@edevil
Copy link
Contributor

edevil commented Oct 10, 2017

"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.

@jamiehannaford
Copy link
Contributor Author

@edevil Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep

@edevil
Copy link
Contributor

edevil commented Oct 10, 2017

Done: #53396
And done: #53642

:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Milestone
v1.8
Development

Successfully merging a pull request may close this issue.

restore e2fsprogs in hyperkube image rphillips/kubernetes
8 participants
@rphillips @edevil @jamiehannaford @ericchiang @ixdy @saad-ali @k8s-github-robot @k8s-ci-robot