Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support labeling of volumes to Pod's SELinux label #15323

Merged
merged 1 commit into from Oct 29, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 12 additions & 1 deletion pkg/kubelet/container/runtime.go
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,8 @@ type Mount struct {
HostPath string
// Whether the mount is read-only.
ReadOnly bool
// Whether the mount needs SELinux relabeling
SELinuxRelabel bool
}

type PortMapping struct {
Expand Down Expand Up @@ -273,7 +275,16 @@ type RunContainerOptions struct {
CgroupParent string
}

type VolumeMap map[string]volume.Volume
// VolumeInfo contains information about the volume.
type VolumeInfo struct {
// Builder is the volume's builder
Builder volume.Builder
// SELinuxLabeled indicates whether this volume has had the
// pod's SELinux label applied to it or not
SELinuxLabeled bool
}

type VolumeMap map[string]VolumeInfo

type Pods []*Pod

Expand Down
23 changes: 20 additions & 3 deletions pkg/kubelet/dockertools/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -606,13 +606,29 @@ func makeEnvList(envs []kubecontainer.EnvVar) (result []string) {
// can be understood by docker.
// Each element in the string is in the form of:
// '<HostPath>:<ContainerPath>', or
// '<HostPath>:<ContainerPath>:ro', if the path is read only.
func makeMountBindings(mounts []kubecontainer.Mount) (result []string) {
// '<HostPath>:<ContainerPath>:ro', if the path is read only, or
// '<HostPath>:<ContainerPath>:Z', if the volume requires SELinux
// relabeling and the pod provides an SELinux label
func makeMountBindings(mounts []kubecontainer.Mount, podHasSELinuxLabel bool) (result []string) {
for _, m := range mounts {
bind := fmt.Sprintf("%s:%s", m.HostPath, m.ContainerPath)
if m.ReadOnly {
bind += ":ro"
}
// Only request relabeling if the pod provides an
// SELinux context. If the pod does not provide an
// SELinux context relabeling will label the volume
// with the container's randomly allocated MCS label.
// This would restrict access to the volume to the
// container which mounts it first.
if m.SELinuxRelabel && podHasSELinuxLabel {
if m.ReadOnly {
bind += ",Z"
} else {
bind += ":Z"
}

}
result = append(result, bind)
}
return
Expand Down Expand Up @@ -766,7 +782,8 @@ func (dm *DockerManager) runContainer(
dm.recorder.Eventf(ref, "Created", "Created with docker id %v", util.ShortenString(dockerContainer.ID, 12))
}

binds := makeMountBindings(opts.Mounts)
podHasSELinuxLabel := pod.Spec.SecurityContext != nil && pod.Spec.SecurityContext.SELinuxOptions != nil
binds := makeMountBindings(opts.Mounts, podHasSELinuxLabel)

// The reason we create and mount the log file in here (not in kubelet) is because
// the file's location depends on the ID of the container, and we need to create and
Expand Down
72 changes: 68 additions & 4 deletions pkg/kubelet/kubelet.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"net/http"
"os"
"path"
"path/filepath"
"sort"
"strings"
"sync"
Expand Down Expand Up @@ -62,6 +63,7 @@ import (
kubeletutil "k8s.io/kubernetes/pkg/kubelet/util"
"k8s.io/kubernetes/pkg/labels"
"k8s.io/kubernetes/pkg/runtime"
"k8s.io/kubernetes/pkg/securitycontext"
"k8s.io/kubernetes/pkg/types"
"k8s.io/kubernetes/pkg/util"
"k8s.io/kubernetes/pkg/util/bandwidth"
Expand All @@ -73,6 +75,7 @@ import (
nodeutil "k8s.io/kubernetes/pkg/util/node"
"k8s.io/kubernetes/pkg/util/oom"
"k8s.io/kubernetes/pkg/util/procfs"
"k8s.io/kubernetes/pkg/util/selinux"
"k8s.io/kubernetes/pkg/util/sets"
"k8s.io/kubernetes/pkg/version"
"k8s.io/kubernetes/pkg/volume"
Expand Down Expand Up @@ -975,6 +978,47 @@ func (kl *Kubelet) syncNodeStatus() {
}
}

// relabelVolumes relabels SELinux volumes to match the pod's
// SELinuxOptions specification. This is only needed if the pod uses
// hostPID or hostIPC. Otherwise relabeling is delegated to docker.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curiosity: if we feel like we know how to do the labelling ourselves, why should we ever defer to docker? Isn't our code net simpler, less coupled, and more fixable if we just do it ourselves?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e.g. what about non-docker runtimes?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thockin I think that we are better off letting docker, or non-docker rumtimes, do the labeling. Because docker is in a better position to know what label the container will run as if one is not provided. In this PR I use the rootContext as a heuristic to decide what the appropriate label 'type' will be. That works well because there are only two answers svirt_sandbox_file_t or "it does not matter because the container is privileged" but if docker changes its defaults we would have to handle that.

In addition I am investigating an issue where containers which run in the hostIPC or hostPID are unconfined by SELinux. That is true with pure docker but not in kube. If I can fix that issue this whole code pathway becomes unnecessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I buy that, but this PR is long overdue, so I'll stop arguing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thockin The chcon runner can definitely be re-used to provide support for rkt and other runtimes. I'll make a follow-up issue.

func (kl *Kubelet) relabelVolumes(pod *api.Pod, volumes kubecontainer.VolumeMap) error {
if pod.Spec.SecurityContext.SELinuxOptions == nil {
return nil
}

rootDirContext, err := kl.getRootDirContext()
if err != nil {
return err
}

chconRunner := selinux.NewChconRunner()
// Apply the pod's Level to the rootDirContext
rootDirSELinuxOptions, err := securitycontext.ParseSELinuxOptions(rootDirContext)
if err != nil {
return err
}

rootDirSELinuxOptions.Level = pod.Spec.SecurityContext.SELinuxOptions.Level
volumeContext := fmt.Sprintf("%s:%s:%s:%s", rootDirSELinuxOptions.User, rootDirSELinuxOptions.Role, rootDirSELinuxOptions.Type, rootDirSELinuxOptions.Level)

for _, volume := range volumes {
if volume.Builder.SupportsSELinux() && !volume.Builder.IsReadOnly() {
// Relabel the volume and its content to match the 'Level' of the pod
err := filepath.Walk(volume.Builder.GetPath(), func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
return chconRunner.SetContext(path, volumeContext)
})
if err != nil {
return err
}
volume.SELinuxLabeled = true
}
}
return nil
}

func makeMounts(pod *api.Pod, podDir string, container *api.Container, podVolumes kubecontainer.VolumeMap) ([]kubecontainer.Mount, error) {
// Kubernetes only mounts on /etc/hosts if :
// - container does not use hostNetwork and
Expand All @@ -991,11 +1035,21 @@ func makeMounts(pod *api.Pod, podDir string, container *api.Container, podVolume
glog.Warningf("Mount cannot be satisified for container %q, because the volume is missing: %q", container.Name, mount)
continue
}

relabelVolume := false
// If the volume supports SELinux and it has not been
// relabeled already and it is not a read-only volume,
// relabel it and mark it as labeled
if vol.Builder.SupportsSELinux() && !vol.SELinuxLabeled && !vol.Builder.IsReadOnly() {
vol.SELinuxLabeled = true
relabelVolume = true
}
mounts = append(mounts, kubecontainer.Mount{
Name: mount.Name,
ContainerPath: mount.MountPath,
HostPath: vol.GetPath(),
ReadOnly: mount.ReadOnly,
Name: mount.Name,
ContainerPath: mount.MountPath,
HostPath: vol.Builder.GetPath(),
ReadOnly: mount.ReadOnly,
SELinuxRelabel: relabelVolume,
})
}
if mountEtcHostsFile {
Expand Down Expand Up @@ -1080,6 +1134,16 @@ func (kl *Kubelet) GenerateRunContainerOptions(pod *api.Pod, container *api.Cont
}

opts.PortMappings = makePortMappings(container)
// Docker does not relabel volumes if the container is running
// in the host pid or ipc namespaces so the kubelet must
// relabel the volumes
if pod.Spec.SecurityContext != nil && (pod.Spec.SecurityContext.HostIPC || pod.Spec.SecurityContext.HostPID) {
err = kl.relabelVolumes(pod, vol)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it a problem that this whole function could fail and the labels are not reverted? I think it is ok, but have to ask.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thockin With this PR it would not be too bad as the next pod will relabel the volume to match its label. So the volume is not rendered unusable. If we change things so that volumes are only relabeled once, only when newly formatted for example, then we might have to handle that. Although TBH I don't think there is value in putting effort into preserving the previous labels; default labels are not usable by pods, and presumably the user does not care about an older pod's labels because the want this pod to use the volume exclusively.

if err != nil {
return nil, err
}
}

opts.Mounts, err = makeMounts(pod, kl.getPodDir(pod.UID), container, vol)
if err != nil {
return nil, err
Expand Down
30 changes: 27 additions & 3 deletions pkg/kubelet/kubelet_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,26 @@ func (f *stubVolume) GetPath() string {
return f.path
}

func (f *stubVolume) IsReadOnly() bool {
return false
}

func (f *stubVolume) SetUp() error {
return nil
}

func (f *stubVolume) SetUpAt(dir string) error {
return nil
}

func (f *stubVolume) SupportsSELinux() bool {
return false
}

func (f *stubVolume) SupportsOwnershipManagement() bool {
return false
}

func TestMakeVolumeMounts(t *testing.T) {
container := api.Container{
VolumeMounts: []api.VolumeMount{
Expand Down Expand Up @@ -537,9 +557,9 @@ func TestMakeVolumeMounts(t *testing.T) {
}

podVolumes := kubecontainer.VolumeMap{
"disk": &stubVolume{"/mnt/disk"},
"disk4": &stubVolume{"/mnt/host"},
"disk5": &stubVolume{"/var/lib/kubelet/podID/volumes/empty/disk5"},
"disk": kubecontainer.VolumeInfo{Builder: &stubVolume{"/mnt/disk"}},
"disk4": kubecontainer.VolumeInfo{Builder: &stubVolume{"/mnt/host"}},
"disk5": kubecontainer.VolumeInfo{Builder: &stubVolume{"/var/lib/kubelet/podID/volumes/empty/disk5"}},
}

pod := api.Pod{
Expand All @@ -558,24 +578,28 @@ func TestMakeVolumeMounts(t *testing.T) {
"/etc/hosts",
"/mnt/disk",
false,
false,
},
{
"disk",
"/mnt/path3",
"/mnt/disk",
true,
false,
},
{
"disk4",
"/mnt/path4",
"/mnt/host",
false,
false,
},
{
"disk5",
"/mnt/path5",
"/var/lib/kubelet/podID/volumes/empty/disk5",
false,
false,
},
}
if !reflect.DeepEqual(mounts, expectedMounts) {
Expand Down
2 changes: 1 addition & 1 deletion pkg/kubelet/rkt/rkt.go
Original file line number Diff line number Diff line change
Expand Up @@ -488,7 +488,7 @@ func (r *Runtime) makePodManifest(pod *api.Pod, pullSecrets []api.Secret) (*appc
manifest.Volumes = append(manifest.Volumes, appctypes.Volume{
Name: *volName,
Kind: "host",
Source: volume.GetPath(),
Source: volume.Builder.GetPath(),
})
}

Expand Down
2 changes: 1 addition & 1 deletion pkg/kubelet/root_context_linux.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (
"github.com/docker/libcontainer/selinux"
)

// getRootContext gets the SELinux context of the kubelet rootDir
// getRootDirContext gets the SELinux context of the kubelet rootDir
// or returns an error.
func (kl *Kubelet) getRootDirContext() (string, error) {
// If SELinux is not enabled, return an empty string
Expand Down
2 changes: 1 addition & 1 deletion pkg/kubelet/volumes.go
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ func (kl *Kubelet) mountExternalVolumes(pod *api.Pod) (kubecontainer.VolumeMap,
return nil, err
}
}
podVolumes[volSpec.Name] = builder
podVolumes[volSpec.Name] = kubecontainer.VolumeInfo{Builder: builder}
}
return podVolumes, nil
}
Expand Down
18 changes: 18 additions & 0 deletions pkg/util/selinux/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/*
Copyright 2015 The Kubernetes Authors All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Package selinux contains selinux utility functions.
package selinux
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

package empty_dir
package selinux

// chconRunner knows how to chcon a directory.
type chconRunner interface {
type ChconRunner interface {
SetContext(dir, context string) error
}

// newChconRunner returns a new chconRunner.
func newChconRunner() chconRunner {
func NewChconRunner() ChconRunner {
return &realChconRunner{}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

package empty_dir
package selinux

import (
"github.com/docker/libcontainer/selinux"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

package empty_dir
package selinux

type realChconRunner struct{}

Expand Down
4 changes: 4 additions & 0 deletions pkg/volume/aws_ebs/aws_ebs.go
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,10 @@ func (b *awsElasticBlockStoreBuilder) IsReadOnly() bool {
return b.readOnly
}

func (b *awsElasticBlockStoreBuilder) SupportsSELinux() bool {
return true
}

func makeGlobalPDPath(host volume.VolumeHost, volumeID string) string {
// Clean up the URI to be more fs-friendly
name := volumeID
Expand Down
4 changes: 4 additions & 0 deletions pkg/volume/cephfs/cephfs.go
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,10 @@ func (cephfsVolume *cephfsBuilder) IsReadOnly() bool {
return cephfsVolume.readonly
}

func (cephfsVolume *cephfsBuilder) SupportsSELinux() bool {
return false
}

type cephfsCleaner struct {
*cephfs
}
Expand Down
4 changes: 4 additions & 0 deletions pkg/volume/cinder/cinder.go
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,10 @@ func (b *cinderVolumeBuilder) IsReadOnly() bool {
return b.readOnly
}

func (b *cinderVolumeBuilder) SupportsSELinux() bool {
return true
}

func makeGlobalPDName(host volume.VolumeHost, devName string) string {
return path.Join(host.GetPluginDir(cinderVolumePluginName), "mounts", devName)
}
Expand Down
4 changes: 4 additions & 0 deletions pkg/volume/downwardapi/downwardapi.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,10 @@ func (d *downwardAPIVolume) IsReadOnly() bool {
return true
}

func (d *downwardAPIVolume) SupportsSELinux() bool {
return true
}

// collectData collects requested downwardAPI in data map.
// Map's key is the requested name of file to dump
// Map's value is the (sorted) content of the field to be dumped in the file.
Expand Down