Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node hostname changes to the name of a k8s daemonset pod #2122

Open
bartelsb opened this issue Mar 11, 2019 · 28 comments
Open

Node hostname changes to the name of a k8s daemonset pod #2122

bartelsb opened this issue Mar 11, 2019 · 28 comments

Comments

@bartelsb
Copy link

@bartelsb bartelsb commented Mar 11, 2019

Migrating issue from kubernetes/kubernetes#70543

Description
A few of the nodes in my cluster changed their transient hostname to the name of the Clamav Pod that is running on them. Clamav mounts / to /host so that it can run scans, but in readonly mode so I don't think that it's directly changing the hostname. Only the transient hostname changed, /etc/hostname stays the same. I can change it back using hostnamectl, but there's no guarantees that it will stay that way. I originally thought it might have something to do with DHCP (that is where my research about transient hostnames changing initially led me), but the dhcpcd client service is not even running on the nodes, and on top of that I don't know how it would get a pod name.

Discovered this issue from an OSSEC (Wazuh) alert.

Steps to reproduce the issue:
I don't know how to reproduce this issue deterministically. I see this happen occasionally (sometimes several per day, sometimes every few days) with clusters running a Clamav daemonset. I am using CoreOS, CRI-O, and k8s 1.11.3.

Describe the results you received:
The hostname of nodes in my cluster changed to the name of pods on those nodes that I'm running with CRI-O.

Describe the results you expected:
The hostname of nodes should not be changed by CRI-O.

Additional information you deem important (e.g. issue happens only occasionally):
Issue occurs at what seem to be random times. Other users reported this happening with other pods, so it doesn't seem to be specific to Clamav (see linked original issue).

Output of crio --version:

crio version 1.11.3
commit: "4fbb0226dd4114aabc5ed13e292179f00e0f8690-dirty"

Additional environment details (AWS, VirtualBox, physical, etc.):

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
AWS
  • OS (e.g. from /etc/os-release):
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1688.5.3
VERSION_ID=1688.5.3
BUILD_ID=2018-04-03-0547
PRETTY_NAME="Container Linux by CoreOS 1688.5.3 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
  • Kernel (e.g. uname -a):
Linux clamav-2dr4d 4.14.32-coreos #1 SMP Tue Apr 3 05:21:26 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux
  • Install tools:
Custom. Kubelet running as systemd unit, all other control plane components are kubelet manifest pods.
@giuseppe

This comment has been minimized.

Copy link
Member

@giuseppe giuseppe commented Mar 13, 2019

is the pod sharing the UTS namespace with the host?

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 13, 2019

Not sure, how would you determine that?

It appears to coincide with the containers restarting and failing to start back up with CreateContainerError. No helpful events when I describe the pod, but this shows up in kubelet logs:

    Mar 13 16:56:04 clamav-npxd9 kubelet[6232]: I0313 16:56:04.471635    6232 server.go:460] Event(v1.ObjectReference{Kind:"Pod", Namespace:"clamav", Name:"clamav-npxd9", UID:"80ae7e5f-3943-11e9-803e-0608d438a58a", APIVersion:"v1", ResourceVersion:"919864", FieldPath:"spec.containers{clamav}"}): type: 'Warning' reason: 'Failed' Error: container create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1713: running lstat on namespace path \"/proc/12304/ns/ipc\" caused \"lstat /proc/12304/ns/ipc: no such file or directory\""
@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 13, 2019

I don't think it's sharing UTS ns for the most part. Under normal operation the container will report the hostname as the name of the pod that it is in, and the physical node will retain its own hostname. This problem only happens occasionally, and only on 1 or 2 pods in the daemonset (out of 6+). But I could definitely see where a problem with creating the new UTS namespace might cause this bug to occur.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 13, 2019

@giuseppe @bhperry For the host network namespace case, we are supposed to use host uts namespace as well. However I think we don't actually do that in CRI-O and it is something that needs fixing. I will test that a bit and see what's happening.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 14, 2019

The Clamav example I've mentioned in the parent issue does not use hostNetwork, but it does have

securityContext: 
  privileged: true

However I have now seen one example where a node took on the transient hostname for a different pod (default-http-backend for ingress-nginx), which is not privileged.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 14, 2019

@bhperry Can you test if for any of the clamav daemonsets the UTS namespace is same as the host?
You can do it like this:

$ ls -l /proc/1/ns/uts 
lrwxrwxrwx. 1 root root 0 Mar 14 12:09 /proc/1/ns/uts -> 'uts:[4026531838]'

$ ls -l /proc/container_pid/ns/uts

If the link points to same value then they are sharing the namespace.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 14, 2019

They are the same on the node that has triggered the bug:

# Container
root@clamav-npxd9:/# ls -l /proc/1/ns/uts
lrwxrwxrwx. 1 root root 0 Mar 14 19:19 /proc/1/ns/uts -> uts:[4026531838]

# Node
clamav-npxd9 ~ # ls -l /proc/1/ns/uts
lrwxrwxrwx. 1 root root 0 Mar 14 19:17 /proc/1/ns/uts -> 'uts:[4026531838]'

And different on a node that has not:

# Container
root@clamav-pg5f4:/# ls -l /proc/1/ns/uts
lrwxrwxrwx. 1 root root 0 Mar 14 19:17 /proc/1/ns/uts -> uts:[4026532531]

# Node
ip-10-240-4-232 ~ # ls -l /proc/1/ns/uts
lrwxrwxrwx. 1 root root 0 Mar 14 18:17 /proc/1/ns/uts -> 'uts:[4026531838]'
@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 14, 2019

@bhperry can you get the config.json for the clamav container from runc?
runc list | grep container_id
The config.json will be under the bundle path.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 14, 2019

{
	"ociVersion": "1.0.0",
	"process": {
		"user": {
			"uid": 0,
			"gid": 0
		},
		"args": [
			"/var/run/clamav/run.sh"
		],
		"env": [
			"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
			"TERM=xterm",
			"HOSTNAME=clamav-npxd9",
			"KUBERNETES_PORT_443_TCP_ADDR=10.32.0.1",
			"KUBERNETES_SERVICE_HOST=10.32.0.1",
			"KUBERNETES_SERVICE_PORT=443",
			"KUBERNETES_SERVICE_PORT_HTTPS=443",
			"KUBERNETES_PORT=tcp://10.32.0.1:443",
			"KUBERNETES_PORT_443_TCP=tcp://10.32.0.1:443",
			"KUBERNETES_PORT_443_TCP_PROTO=tcp",
			"KUBERNETES_PORT_443_TCP_PORT=443",
			"CLAMAV_VERSION=0.100.1"
		],
		"cwd": "/",
		"capabilities": {
			"bounding": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"effective": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"inheritable": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"permitted": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"ambient": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			]
		},
		"oomScoreAdj": -998
	},
	"root": {
		"path": "/var/lib/containers/storage/overlay/e3d7beba2855181d84f876110dfa428f148c538ac22c959fa041cf138e5c7a0a/merged"
	},
	"hostname": "clamav-npxd9",
	"mounts": [
		{
			"destination": "/proc",
			"type": "proc",
			"source": "proc"
		},
		{
			"destination": "/dev",
			"type": "tmpfs",
			"source": "tmpfs",
			"options": [
				"nosuid",
				"strictatime",
				"mode=755",
				"size=65536k"
			]
		},
		{
			"destination": "/dev/pts",
			"type": "devpts",
			"source": "devpts",
			"options": [
				"nosuid",
				"noexec",
				"newinstance",
				"ptmxmode=0666",
				"mode=0620",
				"gid=5"
			]
		},
		{
			"destination": "/dev/mqueue",
			"type": "mqueue",
			"source": "mqueue",
			"options": [
				"nosuid",
				"noexec",
				"nodev"
			]
		},
		{
			"destination": "/sys",
			"type": "sysfs",
			"source": "sysfs",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"rw"
			]
		},
		{
			"destination": "/sys/fs/cgroup",
			"type": "cgroup",
			"source": "cgroup",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"relatime",
				"rw"
			]
		},
		{
			"destination": "/dev/shm",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/6acadc10f144ed4827ff62e5ffe8e8d02287350fea4048ed7b8af4392e9c4b73/userdata/shm",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/etc/resolv.conf",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/6acadc10f144ed4827ff62e5ffe8e8d02287350fea4048ed7b8af4392e9c4b73/userdata/resolv.conf",
			"options": [
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		},
		{
			"destination": "/etc/hostname",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/6acadc10f144ed4827ff62e5ffe8e8d02287350fea4048ed7b8af4392e9c4b73/userdata/hostname",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/host",
			"type": "bind",
			"source": "/",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/etc/clamav",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~configmap/clamav-config",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/root/.aws",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~secret/aws-config",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/etc/hosts",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/etc-hosts",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/dev/termination-log",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/containers/clamav/488d4b79",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/var/log/clamav",
			"type": "bind",
			"source": "/var/log/clamav",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/var/run/secrets/kubernetes.io/serviceaccount",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~secret/clamav-account-token-tr7m9",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		}
	],
	"annotations": {
		"io.kubernetes.container.hash": "31141085",
		"io.kubernetes.container.name": "clamav",
		"io.kubernetes.container.restartCount": "0",
		"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
		"io.kubernetes.container.terminationMessagePolicy": "File",
		"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"31141085\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"File\",\"io.kubernetes.pod.terminationGracePeriod\":\"30\"}",
		"io.kubernetes.cri-o.ContainerID": "99a44f9da85fbd1a38604dd2e613d1c139847c10cbe201d05cb7b203f218873e",
		"io.kubernetes.cri-o.ContainerType": "container",
		"io.kubernetes.cri-o.Created": "2019-03-14T13:25:42.338011225Z",
		"io.kubernetes.cri-o.IP": "10.0.2.13",
		"io.kubernetes.cri-o.Image": "docker.io/datica/clamav@sha256:1ca07ad6e92bfce96b2b5d053cca1095c4c5deb0037793cb936bd04914bafa88",
		"io.kubernetes.cri-o.ImageName": "docker.io/datica/clamav:latest",
		"io.kubernetes.cri-o.ImageRef": "docker.io/datica/clamav@sha256:1ca07ad6e92bfce96b2b5d053cca1095c4c5deb0037793cb936bd04914bafa88",
		"io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"clamav\",\"io.kubernetes.pod.name\":\"clamav-npxd9\",\"io.kubernetes.pod.namespace\":\"clamav\",\"io.kubernetes.pod.uid\":\"80ae7e5f-3943-11e9-803e-0608d438a58a\"}",
		"io.kubernetes.cri-o.LogPath": "/var/log/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/clamav/0.log",
		"io.kubernetes.cri-o.Metadata": "{\"name\":\"clamav\"}",
		"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/e3d7beba2855181d84f876110dfa428f148c538ac22c959fa041cf138e5c7a0a/merged",
		"io.kubernetes.cri-o.Name": "k8s_clamav_clamav-npxd9_clamav_80ae7e5f-3943-11e9-803e-0608d438a58a_0",
		"io.kubernetes.cri-o.ResolvPath": "/var/run/containers/storage/overlay-containers/6acadc10f144ed4827ff62e5ffe8e8d02287350fea4048ed7b8af4392e9c4b73/userdata/resolv.conf",
		"io.kubernetes.cri-o.SandboxID": "6acadc10f144ed4827ff62e5ffe8e8d02287350fea4048ed7b8af4392e9c4b73",
		"io.kubernetes.cri-o.SandboxName": "k8s_POD_clamav-npxd9_clamav_80ae7e5f-3943-11e9-803e-0608d438a58a_3",
		"io.kubernetes.cri-o.SeccompProfilePath": "",
		"io.kubernetes.cri-o.Stdin": "false",
		"io.kubernetes.cri-o.StdinOnce": "false",
		"io.kubernetes.cri-o.TTY": "false",
		"io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/host\",\"host_path\":\"/\",\"readonly\":true},{\"container_path\":\"/etc/clamav\",\"host_path\":\"/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~configmap/clamav-config\",\"readonly\":true},{\"container_path\":\"/var/log/clamav\",\"host_path\":\"/var/log/clamav\",\"readonly\":false},{\"container_path\":\"/root/.aws\",\"host_path\":\"/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~secret/aws-config\",\"readonly\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/volumes/kubernetes.io~secret/clamav-account-token-tr7m9\",\"readonly\":true},{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/etc-hosts\",\"readonly\":false},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/80ae7e5f-3943-11e9-803e-0608d438a58a/containers/clamav/488d4b79\",\"readonly\":false}]",
		"io.kubernetes.pod.name": "clamav-npxd9",
		"io.kubernetes.pod.namespace": "clamav",
		"io.kubernetes.pod.terminationGracePeriod": "30",
		"io.kubernetes.pod.uid": "80ae7e5f-3943-11e9-803e-0608d438a58a"
	},
	"linux": {
		"resources": {
			"devices": [
				{
					"allow": true,
					"access": "rwm"
				}
			],
			"memory": {
				"limit": 1073741824
			},
			"cpu": {
				"shares": 512,
				"quota": 50000,
				"period": 100000
			},
			"pids": {
				"limit": 1024
			}
		},
		"cgroupsPath": "kubepods-pod80ae7e5f_3943_11e9_803e_0608d438a58a.slice:crio:99a44f9da85fbd1a38604dd2e613d1c139847c10cbe201d05cb7b203f218873e",
		"namespaces": [
			{
				"type": "pid"
			},
			{
				"type": "network",
				"path": "/proc/12304/ns/net"
			},
			{
				"type": "ipc",
				"path": "/proc/12304/ns/ipc"
			},
			{
				"type": "uts",
				"path": "/proc/12304/ns/uts"
			},
			{
				"type": "mount"
			}
		],
		"devices": [
			{
				"path": "/dev/autofs",
				"type": "c",
				"major": 10,
				"minor": 235,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/btrfs-control",
				"type": "c",
				"major": 10,
				"minor": 234,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/cpu_dma_latency",
				"type": "c",
				"major": 10,
				"minor": 62,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/cuse",
				"type": "c",
				"major": 10,
				"minor": 203,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/dm-0",
				"type": "b",
				"major": 254,
				"minor": 0,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/full",
				"type": "c",
				"major": 1,
				"minor": 7,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/fuse",
				"type": "c",
				"major": 10,
				"minor": 229,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/hpet",
				"type": "c",
				"major": 10,
				"minor": 228,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/hwrng",
				"type": "c",
				"major": 10,
				"minor": 183,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/input/event0",
				"type": "c",
				"major": 13,
				"minor": 64,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/event1",
				"type": "c",
				"major": 13,
				"minor": 65,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/event2",
				"type": "c",
				"major": 13,
				"minor": 66,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/kmsg",
				"type": "c",
				"major": 1,
				"minor": 11,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/loop-control",
				"type": "c",
				"major": 10,
				"minor": 237,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/mapper/control",
				"type": "c",
				"major": 10,
				"minor": 236,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/mem",
				"type": "c",
				"major": 1,
				"minor": 1,
				"uid": 0,
				"gid": 9
			},
			{
				"path": "/dev/memory_bandwidth",
				"type": "c",
				"major": 10,
				"minor": 59,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/net/tun",
				"type": "c",
				"major": 10,
				"minor": 200,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/network_latency",
				"type": "c",
				"major": 10,
				"minor": 61,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/network_throughput",
				"type": "c",
				"major": 10,
				"minor": 60,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/null",
				"type": "c",
				"major": 1,
				"minor": 3,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme0",
				"type": "c",
				"major": 249,
				"minor": 0,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme0n1",
				"type": "b",
				"major": 259,
				"minor": 2,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p1",
				"type": "b",
				"major": 259,
				"minor": 3,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p2",
				"type": "b",
				"major": 259,
				"minor": 4,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p3",
				"type": "b",
				"major": 259,
				"minor": 5,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p4",
				"type": "b",
				"major": 259,
				"minor": 6,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p6",
				"type": "b",
				"major": 259,
				"minor": 7,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p7",
				"type": "b",
				"major": 259,
				"minor": 8,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme0n1p9",
				"type": "b",
				"major": 259,
				"minor": 9,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme1",
				"type": "c",
				"major": 249,
				"minor": 1,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme1n1",
				"type": "b",
				"major": 259,
				"minor": 0,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme2",
				"type": "c",
				"major": 249,
				"minor": 2,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme2n1",
				"type": "b",
				"major": 259,
				"minor": 1,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme3",
				"type": "c",
				"major": 249,
				"minor": 3,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme3n1",
				"type": "b",
				"major": 259,
				"minor": 10,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme4",
				"type": "c",
				"major": 249,
				"minor": 4,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme4n1",
				"type": "b",
				"major": 259,
				"minor": 11,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/nvme5",
				"type": "c",
				"major": 249,
				"minor": 5,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/nvme5n1",
				"type": "b",
				"major": 259,
				"minor": 12,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/port",
				"type": "c",
				"major": 1,
				"minor": 4,
				"uid": 0,
				"gid": 9
			},
			{
				"path": "/dev/ppp",
				"type": "c",
				"major": 108,
				"minor": 0,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/ptmx",
				"type": "c",
				"major": 5,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/random",
				"type": "c",
				"major": 1,
				"minor": 8,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/rtc0",
				"type": "c",
				"major": 253,
				"minor": 0,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/tty",
				"type": "c",
				"major": 5,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty0",
				"type": "c",
				"major": 4,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty1",
				"type": "c",
				"major": 4,
				"minor": 1,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty10",
				"type": "c",
				"major": 4,
				"minor": 10,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty11",
				"type": "c",
				"major": 4,
				"minor": 11,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty12",
				"type": "c",
				"major": 4,
				"minor": 12,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty13",
				"type": "c",
				"major": 4,
				"minor": 13,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty14",
				"type": "c",
				"major": 4,
				"minor": 14,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty15",
				"type": "c",
				"major": 4,
				"minor": 15,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty16",
				"type": "c",
				"major": 4,
				"minor": 16,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty17",
				"type": "c",
				"major": 4,
				"minor": 17,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty18",
				"type": "c",
				"major": 4,
				"minor": 18,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty19",
				"type": "c",
				"major": 4,
				"minor": 19,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty2",
				"type": "c",
				"major": 4,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty20",
				"type": "c",
				"major": 4,
				"minor": 20,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty21",
				"type": "c",
				"major": 4,
				"minor": 21,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty22",
				"type": "c",
				"major": 4,
				"minor": 22,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty23",
				"type": "c",
				"major": 4,
				"minor": 23,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty24",
				"type": "c",
				"major": 4,
				"minor": 24,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty25",
				"type": "c",
				"major": 4,
				"minor": 25,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty26",
				"type": "c",
				"major": 4,
				"minor": 26,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty27",
				"type": "c",
				"major": 4,
				"minor": 27,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty28",
				"type": "c",
				"major": 4,
				"minor": 28,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty29",
				"type": "c",
				"major": 4,
				"minor": 29,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty3",
				"type": "c",
				"major": 4,
				"minor": 3,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty30",
				"type": "c",
				"major": 4,
				"minor": 30,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty31",
				"type": "c",
				"major": 4,
				"minor": 31,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty32",
				"type": "c",
				"major": 4,
				"minor": 32,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty33",
				"type": "c",
				"major": 4,
				"minor": 33,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty34",
				"type": "c",
				"major": 4,
				"minor": 34,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty35",
				"type": "c",
				"major": 4,
				"minor": 35,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty36",
				"type": "c",
				"major": 4,
				"minor": 36,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty37",
				"type": "c",
				"major": 4,
				"minor": 37,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty38",
				"type": "c",
				"major": 4,
				"minor": 38,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty39",
				"type": "c",
				"major": 4,
				"minor": 39,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty4",
				"type": "c",
				"major": 4,
				"minor": 4,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty40",
				"type": "c",
				"major": 4,
				"minor": 40,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty41",
				"type": "c",
				"major": 4,
				"minor": 41,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty42",
				"type": "c",
				"major": 4,
				"minor": 42,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty43",
				"type": "c",
				"major": 4,
				"minor": 43,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty44",
				"type": "c",
				"major": 4,
				"minor": 44,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty45",
				"type": "c",
				"major": 4,
				"minor": 45,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty46",
				"type": "c",
				"major": 4,
				"minor": 46,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty47",
				"type": "c",
				"major": 4,
				"minor": 47,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty48",
				"type": "c",
				"major": 4,
				"minor": 48,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty49",
				"type": "c",
				"major": 4,
				"minor": 49,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty5",
				"type": "c",
				"major": 4,
				"minor": 5,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty50",
				"type": "c",
				"major": 4,
				"minor": 50,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty51",
				"type": "c",
				"major": 4,
				"minor": 51,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty52",
				"type": "c",
				"major": 4,
				"minor": 52,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty53",
				"type": "c",
				"major": 4,
				"minor": 53,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty54",
				"type": "c",
				"major": 4,
				"minor": 54,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty55",
				"type": "c",
				"major": 4,
				"minor": 55,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty56",
				"type": "c",
				"major": 4,
				"minor": 56,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty57",
				"type": "c",
				"major": 4,
				"minor": 57,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty58",
				"type": "c",
				"major": 4,
				"minor": 58,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty59",
				"type": "c",
				"major": 4,
				"minor": 59,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty6",
				"type": "c",
				"major": 4,
				"minor": 6,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty60",
				"type": "c",
				"major": 4,
				"minor": 60,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty61",
				"type": "c",
				"major": 4,
				"minor": 61,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty62",
				"type": "c",
				"major": 4,
				"minor": 62,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty63",
				"type": "c",
				"major": 4,
				"minor": 63,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty7",
				"type": "c",
				"major": 4,
				"minor": 7,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty8",
				"type": "c",
				"major": 4,
				"minor": 8,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty9",
				"type": "c",
				"major": 4,
				"minor": 9,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/ttyS0",
				"type": "c",
				"major": 4,
				"minor": 64,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/ttyS1",
				"type": "c",
				"major": 4,
				"minor": 65,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyS2",
				"type": "c",
				"major": 4,
				"minor": 66,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyS3",
				"type": "c",
				"major": 4,
				"minor": 67,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyprintk",
				"type": "c",
				"major": 5,
				"minor": 3,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/urandom",
				"type": "c",
				"major": 1,
				"minor": 9,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vcs",
				"type": "c",
				"major": 7,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs1",
				"type": "c",
				"major": 7,
				"minor": 1,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs2",
				"type": "c",
				"major": 7,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs3",
				"type": "c",
				"major": 7,
				"minor": 3,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs4",
				"type": "c",
				"major": 7,
				"minor": 4,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs5",
				"type": "c",
				"major": 7,
				"minor": 5,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs6",
				"type": "c",
				"major": 7,
				"minor": 6,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa",
				"type": "c",
				"major": 7,
				"minor": 128,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa1",
				"type": "c",
				"major": 7,
				"minor": 129,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa2",
				"type": "c",
				"major": 7,
				"minor": 130,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa3",
				"type": "c",
				"major": 7,
				"minor": 131,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa4",
				"type": "c",
				"major": 7,
				"minor": 132,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa5",
				"type": "c",
				"major": 7,
				"minor": 133,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa6",
				"type": "c",
				"major": 7,
				"minor": 134,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vfio/vfio",
				"type": "c",
				"major": 10,
				"minor": 196,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vga_arbiter",
				"type": "c",
				"major": 10,
				"minor": 63,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vhost-net",
				"type": "c",
				"major": 10,
				"minor": 238,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/zero",
				"type": "c",
				"major": 1,
				"minor": 5,
				"uid": 0,
				"gid": 0
			}
		],
		"mountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c238,c868"
	}
}
@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 15, 2019

@bhperry Thanks! To get the last piece of the puzzle, could you get the config.json for the pod container?

crictl inspect clamav_container_id | jq .sandboxId
runc list | grep sandbox_id
@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 15, 2019

No sandboxID via the method you posted, but I got this from the podID of clamav. Is this what you were looking for? Also, unfortunately the container restarted last night and got stuck in CreateContainerError, so I killed it in an attempt to get a containerID. Hopefully that didn't change anything critical in the conf.

{
	"ociVersion": "1.0.0",
	"process": {
		"user": {
			"uid": 0,
			"gid": 0
		},
		"args": [
			"/pause"
		],
		"env": [
			"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
			"TERM=xterm"
		],
		"cwd": "/",
		"capabilities": {
			"bounding": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"effective": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"inheritable": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"permitted": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			]
		},
		"rlimits": [
			{
				"type": "RLIMIT_NOFILE",
				"hard": 1024,
				"soft": 1024
			}
		],
		"oomScoreAdj": -998
	},
	"root": {
		"path": "/var/lib/containers/storage/overlay/245b105d9cd2aca83606657733cb7e1afc8002cfca7e5bfca12ed7b5fc3d6d6f/merged",
		"readonly": true
	},
	"hostname": "clamav-8x4pr",
	"mounts": [
		{
			"destination": "/proc",
			"type": "proc",
			"source": "proc"
		},
		{
			"destination": "/dev",
			"type": "tmpfs",
			"source": "tmpfs",
			"options": [
				"nosuid",
				"strictatime",
				"mode=755",
				"size=65536k"
			]
		},
		{
			"destination": "/dev/pts",
			"type": "devpts",
			"source": "devpts",
			"options": [
				"nosuid",
				"noexec",
				"newinstance",
				"ptmxmode=0666",
				"mode=0620",
				"gid=5"
			]
		},
		{
			"destination": "/dev/mqueue",
			"type": "mqueue",
			"source": "mqueue",
			"options": [
				"nosuid",
				"noexec",
				"nodev"
			]
		},
		{
			"destination": "/sys",
			"type": "sysfs",
			"source": "sysfs",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"ro"
			]
		},
		{
			"destination": "/etc/resolv.conf",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/resolv.conf",
			"options": [
				"ro",
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		},
		{
			"destination": "/dev/shm",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/shm",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/etc/hostname",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/hostname",
			"options": [
				"ro",
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		}
	],
	"annotations": {
		"app": "clamav",
		"controller-revision-hash": "1279576357",
		"io.kubernetes.container.name": "POD",
		"io.kubernetes.cri-o.Annotations": "{\"kubernetes.io/config.seen\":\"2019-03-15T14:43:38.703384913Z\",\"kubernetes.io/config.source\":\"api\"}",
		"io.kubernetes.cri-o.CgroupParent": "kubepods-podb843a827_4730_11e9_803e_0608d438a58a.slice",
		"io.kubernetes.cri-o.ContainerID": "1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d",
		"io.kubernetes.cri-o.ContainerName": "k8s_POD_clamav-8x4pr_clamav_b843a827-4730-11e9-803e-0608d438a58a_0",
		"io.kubernetes.cri-o.ContainerType": "sandbox",
		"io.kubernetes.cri-o.Created": "2019-03-15T14:43:39.056079564Z",
		"io.kubernetes.cri-o.HostName": "clamav-8x4pr",
		"io.kubernetes.cri-o.HostNetwork": "false",
		"io.kubernetes.cri-o.HostnamePath": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/hostname",
		"io.kubernetes.cri-o.KubeName": "clamav-8x4pr",
		"io.kubernetes.cri-o.Labels": "{\"app\":\"clamav\",\"controller-revision-hash\":\"1279576357\",\"io.kubernetes.container.name\":\"POD\",\"io.kubernetes.pod.name\":\"clamav-8x4pr\",\"io.kubernetes.pod.namespace\":\"clamav\",\"io.kubernetes.pod.uid\":\"b843a827-4730-11e9-803e-0608d438a58a\",\"name\":\"clamav\",\"pod-template-generation\":\"1\",\"role\":\"security\"}",
		"io.kubernetes.cri-o.LogPath": "/var/log/pods/b843a827-4730-11e9-803e-0608d438a58a/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d.log",
		"io.kubernetes.cri-o.Metadata": "{\"name\":\"clamav-8x4pr\",\"uid\":\"b843a827-4730-11e9-803e-0608d438a58a\",\"namespace\":\"clamav\"}",
		"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/245b105d9cd2aca83606657733cb7e1afc8002cfca7e5bfca12ed7b5fc3d6d6f/merged",
		"io.kubernetes.cri-o.Name": "k8s_clamav-8x4pr_clamav_b843a827-4730-11e9-803e-0608d438a58a_0",
		"io.kubernetes.cri-o.Namespace": "clamav",
		"io.kubernetes.cri-o.NamespaceOptions": "{\"pid\":1}",
		"io.kubernetes.cri-o.PortMappings": "null",
		"io.kubernetes.cri-o.PrivilegedRuntime": "true",
		"io.kubernetes.cri-o.ResolvPath": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/resolv.conf",
		"io.kubernetes.cri-o.SandboxID": "1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d",
		"io.kubernetes.cri-o.SeccompProfilePath": "",
		"io.kubernetes.cri-o.ShmPath": "/var/run/containers/storage/overlay-containers/1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d/userdata/shm",
		"io.kubernetes.cri-o.TrustedSandbox": "true",
		"io.kubernetes.pod.name": "clamav-8x4pr",
		"io.kubernetes.pod.namespace": "clamav",
		"io.kubernetes.pod.uid": "b843a827-4730-11e9-803e-0608d438a58a",
		"kubernetes.io/config.seen": "2019-03-15T14:43:38.703384913Z",
		"kubernetes.io/config.source": "api",
		"name": "clamav",
		"pod-template-generation": "1",
		"role": "security"
	},
	"linux": {
		"resources": {
			"devices": [
				{
					"allow": false,
					"access": "rwm"
				}
			],
			"cpu": {
				"shares": 2
			}
		},
		"cgroupsPath": "kubepods-podb843a827_4730_11e9_803e_0608d438a58a.slice:crio:1a4cf2a7b694d82547bd3e918244ad19de0df04d30d53afd890db7549a09847d",
		"namespaces": [
			{
				"type": "pid"
			},
			{
				"type": "network"
			},
			{
				"type": "ipc"
			},
			{
				"type": "uts"
			},
			{
				"type": "mount"
			}
		],
		"seccomp": {
			"defaultAction": "SCMP_ACT_ERRNO",
			"architectures": [
				"SCMP_ARCH_X86_64",
				"SCMP_ARCH_X86",
				"SCMP_ARCH_X32"
			],
			"syscalls": [
				{
					"names": [
						"accept",
						"accept4",
						"access",
						"alarm",
						"bind",
						"brk",
						"capget",
						"capset",
						"chdir",
						"chmod",
						"chown",
						"chown32",
						"clock_getres",
						"clock_gettime",
						"clock_nanosleep",
						"close",
						"connect",
						"copy_file_range",
						"creat",
						"dup",
						"dup2",
						"dup3",
						"epoll_create",
						"epoll_create1",
						"epoll_ctl",
						"epoll_ctl_old",
						"epoll_pwait",
						"epoll_wait",
						"epoll_wait_old",
						"eventfd",
						"eventfd2",
						"execve",
						"execveat",
						"exit",
						"exit_group",
						"faccessat",
						"fadvise64",
						"fadvise64_64",
						"fallocate",
						"fanotify_mark",
						"fchdir",
						"fchmod",
						"fchmodat",
						"fchown",
						"fchown32",
						"fchownat",
						"fcntl",
						"fcntl64",
						"fdatasync",
						"fgetxattr",
						"flistxattr",
						"flock",
						"fork",
						"fremovexattr",
						"fsetxattr",
						"fstat",
						"fstat64",
						"fstatat64",
						"fstatfs",
						"fstatfs64",
						"fsync",
						"ftruncate",
						"ftruncate64",
						"futex",
						"futimesat",
						"getcpu",
						"getcwd",
						"getdents",
						"getdents64",
						"getegid",
						"getegid32",
						"geteuid",
						"geteuid32",
						"getgid",
						"getgid32",
						"getgroups",
						"getgroups32",
						"getitimer",
						"getpeername",
						"getpgid",
						"getpgrp",
						"getpid",
						"getppid",
						"getpriority",
						"getrandom",
						"getresgid",
						"getresgid32",
						"getresuid",
						"getresuid32",
						"getrlimit",
						"get_robust_list",
						"getrusage",
						"getsid",
						"getsockname",
						"getsockopt",
						"get_thread_area",
						"gettid",
						"gettimeofday",
						"getuid",
						"getuid32",
						"getxattr",
						"inotify_add_watch",
						"inotify_init",
						"inotify_init1",
						"inotify_rm_watch",
						"io_cancel",
						"ioctl",
						"io_destroy",
						"io_getevents",
						"ioprio_get",
						"ioprio_set",
						"io_setup",
						"io_submit",
						"ipc",
						"kill",
						"lchown",
						"lchown32",
						"lgetxattr",
						"link",
						"linkat",
						"listen",
						"listxattr",
						"llistxattr",
						"_llseek",
						"lremovexattr",
						"lseek",
						"lsetxattr",
						"lstat",
						"lstat64",
						"madvise",
						"memfd_create",
						"mincore",
						"mkdir",
						"mkdirat",
						"mknod",
						"mknodat",
						"mlock",
						"mlock2",
						"mlockall",
						"mmap",
						"mmap2",
						"mprotect",
						"mq_getsetattr",
						"mq_notify",
						"mq_open",
						"mq_timedreceive",
						"mq_timedsend",
						"mq_unlink",
						"mremap",
						"msgctl",
						"msgget",
						"msgrcv",
						"msgsnd",
						"msync",
						"munlock",
						"munlockall",
						"munmap",
						"nanosleep",
						"newfstatat",
						"_newselect",
						"open",
						"openat",
						"pause",
						"pipe",
						"pipe2",
						"poll",
						"ppoll",
						"prctl",
						"pread64",
						"preadv",
						"prlimit64",
						"pselect6",
						"pwrite64",
						"pwritev",
						"read",
						"readahead",
						"readlink",
						"readlinkat",
						"readv",
						"recv",
						"recvfrom",
						"recvmmsg",
						"recvmsg",
						"remap_file_pages",
						"removexattr",
						"rename",
						"renameat",
						"renameat2",
						"restart_syscall",
						"rmdir",
						"rt_sigaction",
						"rt_sigpending",
						"rt_sigprocmask",
						"rt_sigqueueinfo",
						"rt_sigreturn",
						"rt_sigsuspend",
						"rt_sigtimedwait",
						"rt_tgsigqueueinfo",
						"sched_getaffinity",
						"sched_getattr",
						"sched_getparam",
						"sched_get_priority_max",
						"sched_get_priority_min",
						"sched_getscheduler",
						"sched_rr_get_interval",
						"sched_setaffinity",
						"sched_setattr",
						"sched_setparam",
						"sched_setscheduler",
						"sched_yield",
						"seccomp",
						"select",
						"semctl",
						"semget",
						"semop",
						"semtimedop",
						"send",
						"sendfile",
						"sendfile64",
						"sendmmsg",
						"sendmsg",
						"sendto",
						"setfsgid",
						"setfsgid32",
						"setfsuid",
						"setfsuid32",
						"setgid",
						"setgid32",
						"setgroups",
						"setgroups32",
						"setitimer",
						"setpgid",
						"setpriority",
						"setregid",
						"setregid32",
						"setresgid",
						"setresgid32",
						"setresuid",
						"setresuid32",
						"setreuid",
						"setreuid32",
						"setrlimit",
						"set_robust_list",
						"setsid",
						"setsockopt",
						"set_thread_area",
						"set_tid_address",
						"setuid",
						"setuid32",
						"setxattr",
						"shmat",
						"shmctl",
						"shmdt",
						"shmget",
						"shutdown",
						"sigaltstack",
						"signalfd",
						"signalfd4",
						"sigreturn",
						"socket",
						"socketcall",
						"socketpair",
						"splice",
						"stat",
						"stat64",
						"statfs",
						"statfs64",
						"symlink",
						"symlinkat",
						"sync",
						"sync_file_range",
						"syncfs",
						"sysinfo",
						"syslog",
						"tee",
						"tgkill",
						"time",
						"timer_create",
						"timer_delete",
						"timerfd_create",
						"timerfd_gettime",
						"timerfd_settime",
						"timer_getoverrun",
						"timer_gettime",
						"timer_settime",
						"times",
						"tkill",
						"truncate",
						"truncate64",
						"ugetrlimit",
						"umask",
						"uname",
						"unlink",
						"unlinkat",
						"utime",
						"utimensat",
						"utimes",
						"vfork",
						"vmsplice",
						"wait4",
						"waitid",
						"waitpid",
						"write",
						"writev"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"personality"
					],
					"action": "SCMP_ACT_ALLOW",
					"args": [
						{
							"index": 0,
							"value": 0,
							"op": "SCMP_CMP_EQ"
						},
						{
							"index": 0,
							"value": 8,
							"op": "SCMP_CMP_EQ"
						},
						{
							"index": 0,
							"value": 4294967295,
							"op": "SCMP_CMP_EQ"
						}
					]
				},
				{
					"names": [
						"chroot"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"clone"
					],
					"action": "SCMP_ACT_ALLOW",
					"args": [
						{
							"index": 0,
							"value": 2080505856,
							"op": "SCMP_CMP_MASKED_EQ"
						}
					]
				},
				{
					"names": [
						"arch_prctl"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"modify_ldt"
					],
					"action": "SCMP_ACT_ALLOW"
				}
			]
		},
		"mountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c640,c974"
	}
}
@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 15, 2019

Let me know if you need the config from a container that is currently triggering the bug. There's a few clusters that have this problem which I could grab a config from.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 15, 2019

@bhperry Yes, I need new configs from where you are seeing the bug. Both the container and the pod/sandbox config.json.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 15, 2019

Sure thing. Here's container:

ClamAV Container Config

{
	"ociVersion": "1.0.0",
	"process": { 
		"user": {
			"uid": 0,
			"gid": 0
		},
		"args": [
			"/var/run/clamav/run.sh"
		],
		"env": [
			"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
			"TERM=xterm",
			"HOSTNAME=clamav-t95hr",
			"KUBERNETES_PORT_443_TCP_PORT=443",
			"KUBERNETES_PORT_443_TCP_ADDR=10.32.0.1",
			"KUBERNETES_SERVICE_HOST=10.32.0.1",
			"KUBERNETES_SERVICE_PORT=443",
			"KUBERNETES_SERVICE_PORT_HTTPS=443",
			"KUBERNETES_PORT=tcp://10.32.0.1:443",
			"KUBERNETES_PORT_443_TCP=tcp://10.32.0.1:443",
			"KUBERNETES_PORT_443_TCP_PROTO=tcp",
			"CLAMAV_VERSION=0.100.1"
		],
		"cwd": "/",
		"capabilities": {
			"bounding": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"effective": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"inheritable": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"permitted": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			],
			"ambient": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_DAC_READ_SEARCH",
				"CAP_FOWNER",
				"CAP_FSETID",
				"CAP_KILL",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_LINUX_IMMUTABLE",
				"CAP_NET_BIND_SERVICE",
				"CAP_NET_BROADCAST",
				"CAP_NET_ADMIN",
				"CAP_NET_RAW",
				"CAP_IPC_LOCK",
				"CAP_IPC_OWNER",
				"CAP_SYS_MODULE",
				"CAP_SYS_RAWIO",
				"CAP_SYS_CHROOT",
				"CAP_SYS_PTRACE",
				"CAP_SYS_PACCT",
				"CAP_SYS_ADMIN",
				"CAP_SYS_BOOT",
				"CAP_SYS_NICE",
				"CAP_SYS_RESOURCE",
				"CAP_SYS_TIME",
				"CAP_SYS_TTY_CONFIG",
				"CAP_MKNOD",
				"CAP_LEASE",
				"CAP_AUDIT_WRITE",
				"CAP_AUDIT_CONTROL",
				"CAP_SETFCAP",
				"CAP_MAC_OVERRIDE",
				"CAP_MAC_ADMIN",
				"CAP_SYSLOG",
				"CAP_WAKE_ALARM",
				"CAP_BLOCK_SUSPEND",
				"CAP_AUDIT_READ"
			]
		},
		"oomScoreAdj": -998
	},
	"root": {
		"path": "/var/lib/containers/storage/overlay/4a81ff6a2bb60cf96a2c0063dd12d635365eb92e639679304ba1d6f0d90bc984/merged"
	},
	"hostname": "clamav-t95hr",
	"mounts": [
		{
			"destination": "/proc",
			"type": "proc",
			"source": "proc"
		},
		{
			"destination": "/dev",
			"type": "tmpfs",
			"source": "tmpfs",
			"options": [
				"nosuid",
				"strictatime",
				"mode=755",
				"size=65536k"
			]
		},
		{
			"destination": "/dev/pts",
			"type": "devpts",
			"source": "devpts",
			"options": [
				"nosuid",
				"noexec",
				"newinstance",
				"ptmxmode=0666",
				"mode=0620",
				"gid=5"
			]
		},
		{
			"destination": "/dev/mqueue",
			"type": "mqueue",
			"source": "mqueue",
			"options": [
				"nosuid",
				"noexec",
				"nodev"
			]
		},
		{
			"destination": "/sys",
			"type": "sysfs",
			"source": "sysfs",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"rw"
			]
		},
		{
			"destination": "/sys/fs/cgroup",
			"type": "cgroup",
			"source": "cgroup",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"relatime",
				"rw"
			]
		},
		{
			"destination": "/dev/shm",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/shm",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/etc/resolv.conf",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/resolv.conf",
			"options": [
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		},
		{
			"destination": "/etc/hostname",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/hostname",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/host",
			"type": "bind",
			"source": "/",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/etc/clamav",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~configmap/clamav-config",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/root/.aws",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~secret/aws-config",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/etc/hosts",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/etc-hosts",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/dev/termination-log",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/containers/clamav/87031f08",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/var/log/clamav",
			"type": "bind",
			"source": "/var/log/clamav",
			"options": [
				"rw",
				"rbind",
				"rprivate",
				"bind"
			]
		},
		{
			"destination": "/var/run/secrets/kubernetes.io/serviceaccount",
			"type": "bind",
			"source": "/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~secret/clamav-account-token-ws9nl",
			"options": [
				"ro",
				"rbind",
				"rprivate",
				"bind"
			]
		}
	],
	"annotations": {
		"io.kubernetes.container.hash": "3fe673ab",
		"io.kubernetes.container.name": "clamav",
		"io.kubernetes.container.restartCount": "0",
		"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
		"io.kubernetes.container.terminationMessagePolicy": "File",
		"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"3fe673ab\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"File\",\"io.kubernetes.pod.terminationGracePeriod\":\"30\"}",
		"io.kubernetes.cri-o.ContainerID": "5e39835a111ab29c12c98f99fe1ea60d79c5acb236ff1b0adfba85b9d2f3f356",
		"io.kubernetes.cri-o.ContainerType": "container",
		"io.kubernetes.cri-o.Created": "2019-03-15T12:56:19.263422621Z",
		"io.kubernetes.cri-o.IP": "10.0.2.19",
		"io.kubernetes.cri-o.Image": "docker.io/datica/clamav@sha256:1ca07ad6e92bfce96b2b5d053cca1095c4c5deb0037793cb936bd04914bafa88",
		"io.kubernetes.cri-o.ImageName": "docker.io/datica/clamav:latest",
		"io.kubernetes.cri-o.ImageRef": "docker.io/datica/clamav@sha256:1ca07ad6e92bfce96b2b5d053cca1095c4c5deb0037793cb936bd04914bafa88",
		"io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"clamav\",\"io.kubernetes.pod.name\":\"clamav-t95hr\",\"io.kubernetes.pod.namespace\":\"clamav\",\"io.kubernetes.pod.uid\":\"682811c3-f42c-11e8-8dc8-0226acc80e42\"}",
		"io.kubernetes.cri-o.LogPath": "/var/log/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/clamav/0.log",
		"io.kubernetes.cri-o.Metadata": "{\"name\":\"clamav\"}",
		"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/4a81ff6a2bb60cf96a2c0063dd12d635365eb92e639679304ba1d6f0d90bc984/merged",
		"io.kubernetes.cri-o.Name": "k8s_clamav_clamav-t95hr_clamav_682811c3-f42c-11e8-8dc8-0226acc80e42_0",
		"io.kubernetes.cri-o.ResolvPath": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/resolv.conf",
		"io.kubernetes.cri-o.SandboxID": "d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0",
		"io.kubernetes.cri-o.SandboxName": "k8s_POD_clamav-t95hr_clamav_682811c3-f42c-11e8-8dc8-0226acc80e42_6",
		"io.kubernetes.cri-o.SeccompProfilePath": "",
		"io.kubernetes.cri-o.Stdin": "false",
		"io.kubernetes.cri-o.StdinOnce": "false",
		"io.kubernetes.cri-o.TTY": "false",
		"io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/host\",\"host_path\":\"/\",\"readonly\":true},{\"container_path\":\"/etc/clamav\",\"host_path\":\"/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~configmap/clamav-config\",\"readonly\":true},{\"container_path\":\"/var/log/clamav\",\"host_path\":\"/var/log/clamav\",\"readonly\":false},{\"container_path\":\"/root/.aws\",\"host_path\":\"/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~secret/aws-config\",\"readonly\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/volumes/kubernetes.io~secret/clamav-account-token-ws9nl\",\"readonly\":true},{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/etc-hosts\",\"readonly\":false},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/containers/clamav/87031f08\",\"readonly\":false}]",
		"io.kubernetes.pod.name": "clamav-t95hr",
		"io.kubernetes.pod.namespace": "clamav",
		"io.kubernetes.pod.terminationGracePeriod": "30",
		"io.kubernetes.pod.uid": "682811c3-f42c-11e8-8dc8-0226acc80e42"
	},
	"linux": {
		"resources": {
			"devices": [
				{
					"allow": true,
					"access": "rwm"
				}
			],
			"memory": {
				"limit": 1073741824
			},
			"cpu": {
				"shares": 512,
				"quota": 50000,
				"period": 100000
			},
			"pids": {
				"limit": 1024
			}
		},
		"cgroupsPath": "kubepods-pod682811c3_f42c_11e8_8dc8_0226acc80e42.slice:crio:5e39835a111ab29c12c98f99fe1ea60d79c5acb236ff1b0adfba85b9d2f3f356",
		"namespaces": [
			{
				"type": "pid"
			},
			{
				"type": "network",
				"path": "/proc/17425/ns/net"
			},
			{
				"type": "ipc",
				"path": "/proc/17425/ns/ipc"
			},
			{
				"type": "uts",
				"path": "/proc/17425/ns/uts"
			},
			{
				"type": "mount"
			}
		],
		"devices": [
			{
				"path": "/dev/autofs",
				"type": "c",
				"major": 10,
				"minor": 235,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/btrfs-control",
				"type": "c",
				"major": 10,
				"minor": 234,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/cpu_dma_latency",
				"type": "c",
				"major": 10,
				"minor": 61,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/cuse",
				"type": "c",
				"major": 10,
				"minor": 203,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/dm-0",
				"type": "b",
				"major": 254,
				"minor": 0,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/full",
				"type": "c",
				"major": 1,
				"minor": 7,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/fuse",
				"type": "c",
				"major": 10,
				"minor": 229,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/hpet",
				"type": "c",
				"major": 10,
				"minor": 228,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/hwrng",
				"type": "c",
				"major": 10,
				"minor": 183,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/input/event0",
				"type": "c",
				"major": 13,
				"minor": 64,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/event1",
				"type": "c",
				"major": 13,
				"minor": 65,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/event2",
				"type": "c",
				"major": 13,
				"minor": 66,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/event3",
				"type": "c",
				"major": 13,
				"minor": 67,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/mice",
				"type": "c",
				"major": 13,
				"minor": 63,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/input/mouse0",
				"type": "c",
				"major": 13,
				"minor": 32,
				"uid": 0,
				"gid": 28
			},
			{
				"path": "/dev/kmsg",
				"type": "c",
				"major": 1,
				"minor": 11,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/loop-control",
				"type": "c",
				"major": 10,
				"minor": 237,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/mapper/control",
				"type": "c",
				"major": 10,
				"minor": 236,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/mem",
				"type": "c",
				"major": 1,
				"minor": 1,
				"uid": 0,
				"gid": 9
			},
			{
				"path": "/dev/memory_bandwidth",
				"type": "c",
				"major": 10,
				"minor": 58,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/net/tun",
				"type": "c",
				"major": 10,
				"minor": 200,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/network_latency",
				"type": "c",
				"major": 10,
				"minor": 60,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/network_throughput",
				"type": "c",
				"major": 10,
				"minor": 59,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/null",
				"type": "c",
				"major": 1,
				"minor": 3,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/port",
				"type": "c",
				"major": 1,
				"minor": 4,
				"uid": 0,
				"gid": 9
			},
			{
				"path": "/dev/ppp",
				"type": "c",
				"major": 108,
				"minor": 0,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/ptmx",
				"type": "c",
				"major": 5,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/random",
				"type": "c",
				"major": 1,
				"minor": 8,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/rtc0",
				"type": "c",
				"major": 253,
				"minor": 0,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/tty",
				"type": "c",
				"major": 5,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty0",
				"type": "c",
				"major": 4,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty1",
				"type": "c",
				"major": 4,
				"minor": 1,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty10",
				"type": "c",
				"major": 4,
				"minor": 10,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty11",
				"type": "c",
				"major": 4,
				"minor": 11,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty12",
				"type": "c",
				"major": 4,
				"minor": 12,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty13",
				"type": "c",
				"major": 4,
				"minor": 13,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty14",
				"type": "c",
				"major": 4,
				"minor": 14,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty15",
				"type": "c",
				"major": 4,
				"minor": 15,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty16",
				"type": "c",
				"major": 4,
				"minor": 16,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty17",
				"type": "c",
				"major": 4,
				"minor": 17,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty18",
				"type": "c",
				"major": 4,
				"minor": 18,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty19",
				"type": "c",
				"major": 4,
				"minor": 19,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty2",
				"type": "c",
				"major": 4,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty20",
				"type": "c",
				"major": 4,
				"minor": 20,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty21",
				"type": "c",
				"major": 4,
				"minor": 21,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty22",
				"type": "c",
				"major": 4,
				"minor": 22,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty23",
				"type": "c",
				"major": 4,
				"minor": 23,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty24",
				"type": "c",
				"major": 4,
				"minor": 24,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty25",
				"type": "c",
				"major": 4,
				"minor": 25,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty26",
				"type": "c",
				"major": 4,
				"minor": 26,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty27",
				"type": "c",
				"major": 4,
				"minor": 27,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty28",
				"type": "c",
				"major": 4,
				"minor": 28,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty29",
				"type": "c",
				"major": 4,
				"minor": 29,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty3",
				"type": "c",
				"major": 4,
				"minor": 3,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty30",
				"type": "c",
				"major": 4,
				"minor": 30,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty31",
				"type": "c",
				"major": 4,
				"minor": 31,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty32",
				"type": "c",
				"major": 4,
				"minor": 32,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty33",
				"type": "c",
				"major": 4,
				"minor": 33,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty34",
				"type": "c",
				"major": 4,
				"minor": 34,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty35",
				"type": "c",
				"major": 4,
				"minor": 35,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty36",
				"type": "c",
				"major": 4,
				"minor": 36,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty37",
				"type": "c",
				"major": 4,
				"minor": 37,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty38",
				"type": "c",
				"major": 4,
				"minor": 38,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty39",
				"type": "c",
				"major": 4,
				"minor": 39,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty4",
				"type": "c",
				"major": 4,
				"minor": 4,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty40",
				"type": "c",
				"major": 4,
				"minor": 40,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty41",
				"type": "c",
				"major": 4,
				"minor": 41,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty42",
				"type": "c",
				"major": 4,
				"minor": 42,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty43",
				"type": "c",
				"major": 4,
				"minor": 43,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty44",
				"type": "c",
				"major": 4,
				"minor": 44,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty45",
				"type": "c",
				"major": 4,
				"minor": 45,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty46",
				"type": "c",
				"major": 4,
				"minor": 46,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty47",
				"type": "c",
				"major": 4,
				"minor": 47,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty48",
				"type": "c",
				"major": 4,
				"minor": 48,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty49",
				"type": "c",
				"major": 4,
				"minor": 49,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty5",
				"type": "c",
				"major": 4,
				"minor": 5,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty50",
				"type": "c",
				"major": 4,
				"minor": 50,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty51",
				"type": "c",
				"major": 4,
				"minor": 51,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty52",
				"type": "c",
				"major": 4,
				"minor": 52,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty53",
				"type": "c",
				"major": 4,
				"minor": 53,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty54",
				"type": "c",
				"major": 4,
				"minor": 54,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty55",
				"type": "c",
				"major": 4,
				"minor": 55,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty56",
				"type": "c",
				"major": 4,
				"minor": 56,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty57",
				"type": "c",
				"major": 4,
				"minor": 57,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty58",
				"type": "c",
				"major": 4,
				"minor": 58,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty59",
				"type": "c",
				"major": 4,
				"minor": 59,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty6",
				"type": "c",
				"major": 4,
				"minor": 6,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty60",
				"type": "c",
				"major": 4,
				"minor": 60,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty61",
				"type": "c",
				"major": 4,
				"minor": 61,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty62",
				"type": "c",
				"major": 4,
				"minor": 62,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty63",
				"type": "c",
				"major": 4,
				"minor": 63,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty7",
				"type": "c",
				"major": 4,
				"minor": 7,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty8",
				"type": "c",
				"major": 4,
				"minor": 8,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/tty9",
				"type": "c",
				"major": 4,
				"minor": 9,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/ttyS0",
				"type": "c",
				"major": 4,
				"minor": 64,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/ttyS1",
				"type": "c",
				"major": 4,
				"minor": 65,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyS2",
				"type": "c",
				"major": 4,
				"minor": 66,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyS3",
				"type": "c",
				"major": 4,
				"minor": 67,
				"uid": 0,
				"gid": 249
			},
			{
				"path": "/dev/ttyprintk",
				"type": "c",
				"major": 5,
				"minor": 3,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/urandom",
				"type": "c",
				"major": 1,
				"minor": 9,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vcs",
				"type": "c",
				"major": 7,
				"minor": 0,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs1",
				"type": "c",
				"major": 7,
				"minor": 1,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs2",
				"type": "c",
				"major": 7,
				"minor": 2,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs3",
				"type": "c",
				"major": 7,
				"minor": 3,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs4",
				"type": "c",
				"major": 7,
				"minor": 4,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs5",
				"type": "c",
				"major": 7,
				"minor": 5,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcs6",
				"type": "c",
				"major": 7,
				"minor": 6,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa",
				"type": "c",
				"major": 7,
				"minor": 128,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa1",
				"type": "c",
				"major": 7,
				"minor": 129,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa2",
				"type": "c",
				"major": 7,
				"minor": 130,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa3",
				"type": "c",
				"major": 7,
				"minor": 131,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa4",
				"type": "c",
				"major": 7,
				"minor": 132,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa5",
				"type": "c",
				"major": 7,
				"minor": 133,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsa6",
				"type": "c",
				"major": 7,
				"minor": 134,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu",
				"type": "c",
				"major": 7,
				"minor": 64,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu1",
				"type": "c",
				"major": 7,
				"minor": 65,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu2",
				"type": "c",
				"major": 7,
				"minor": 66,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu3",
				"type": "c",
				"major": 7,
				"minor": 67,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu4",
				"type": "c",
				"major": 7,
				"minor": 68,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu5",
				"type": "c",
				"major": 7,
				"minor": 69,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vcsu6",
				"type": "c",
				"major": 7,
				"minor": 70,
				"uid": 0,
				"gid": 5
			},
			{
				"path": "/dev/vfio/vfio",
				"type": "c",
				"major": 10,
				"minor": 196,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vga_arbiter",
				"type": "c",
				"major": 10,
				"minor": 63,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/vhost-net",
				"type": "c",
				"major": 10,
				"minor": 238,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/xen/hypercall",
				"type": "c",
				"major": 10,
				"minor": 56,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/xen/privcmd",
				"type": "c",
				"major": 10,
				"minor": 57,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/xen/xenbus",
				"type": "c",
				"major": 10,
				"minor": 62,
				"uid": 0,
				"gid": 0
			},
			{
				"path": "/dev/xvda",
				"type": "b",
				"major": 202,
				"minor": 0,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda1",
				"type": "b",
				"major": 202,
				"minor": 1,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda2",
				"type": "b",
				"major": 202,
				"minor": 2,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda3",
				"type": "b",
				"major": 202,
				"minor": 3,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda4",
				"type": "b",
				"major": 202,
				"minor": 4,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda6",
				"type": "b",
				"major": 202,
				"minor": 6,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda7",
				"type": "b",
				"major": 202,
				"minor": 7,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvda9",
				"type": "b",
				"major": 202,
				"minor": 9,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvdb",
				"type": "b",
				"major": 202,
				"minor": 16,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvdbp",
				"type": "b",
				"major": 202,
				"minor": 17152,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvdc",
				"type": "b",
				"major": 202,
				"minor": 32,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvdd",
				"type": "b",
				"major": 202,
				"minor": 48,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/xvde",
				"type": "b",
				"major": 202,
				"minor": 64,
				"uid": 0,
				"gid": 6
			},
			{
				"path": "/dev/zero",
				"type": "c",
				"major": 1,
				"minor": 5,
				"uid": 0,
				"gid": 0
			}
		],
		"mountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c45,c386"
	}
}

And pod

Clamav Pod Config

{
	"ociVersion": "1.0.0",
	"process": {
		"user": {
			"uid": 0,
			"gid": 0
		},
		"args": [
			"/pause"
		],
		"env": [
			"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
			"TERM=xterm"
		],
		"cwd": "/",
		"capabilities": {
			"bounding": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"effective": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"inheritable": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			],
			"permitted": [
				"CAP_CHOWN",
				"CAP_DAC_OVERRIDE",
				"CAP_FSETID",
				"CAP_FOWNER",
				"CAP_NET_RAW",
				"CAP_SETGID",
				"CAP_SETUID",
				"CAP_SETPCAP",
				"CAP_NET_BIND_SERVICE",
				"CAP_SYS_CHROOT",
				"CAP_KILL"
			]
		},
		"rlimits": [
			{
				"type": "RLIMIT_NOFILE",
				"hard": 1024,
				"soft": 1024
			}
		],
		"oomScoreAdj": -998
	},
	"root": {
		"path": "/var/lib/containers/storage/overlay/92cbed9223a085575a9f3cf7e5dcf6ea7215e8f8f89d6a7f8a1246a0ed4e6523/merged",
		"readonly": true
	},
	"hostname": "clamav-t95hr",
	"mounts": [
		{
			"destination": "/proc",
			"type": "proc",
			"source": "proc"
		},
		{
			"destination": "/dev",
			"type": "tmpfs",
			"source": "tmpfs",
			"options": [
				"nosuid",
				"strictatime",
				"mode=755",
				"size=65536k"
			]
		},
		{
			"destination": "/dev/pts",
			"type": "devpts",
			"source": "devpts",
			"options": [
				"nosuid",
				"noexec",
				"newinstance",
				"ptmxmode=0666",
				"mode=0620",
				"gid=5"
			]
		},
		{
			"destination": "/dev/mqueue",
			"type": "mqueue",
			"source": "mqueue",
			"options": [
				"nosuid",
				"noexec",
				"nodev"
			]
		},
		{
			"destination": "/sys",
			"type": "sysfs",
			"source": "sysfs",
			"options": [
				"nosuid",
				"noexec",
				"nodev",
				"ro"
			]
		},
		{
			"destination": "/etc/resolv.conf",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/resolv.conf",
			"options": [
				"ro",
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		},
		{
			"destination": "/dev/shm",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/shm",
			"options": [
				"rw",
				"bind"
			]
		},
		{
			"destination": "/etc/hostname",
			"type": "bind",
			"source": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/hostname",
			"options": [
				"ro",
				"bind",
				"nodev",
				"nosuid",
				"noexec"
			]
		}
	],
	"annotations": {
		"app": "clamav",
		"controller-revision-hash": "1279576357",
		"io.kubernetes.container.name": "POD",
		"io.kubernetes.cri-o.Annotations": "{\"kubernetes.io/config.seen\":\"2019-02-28T16:54:30.880812155Z\",\"kubernetes.io/config.source\":\"api\"}",
		"io.kubernetes.cri-o.CgroupParent": "kubepods-pod682811c3_f42c_11e8_8dc8_0226acc80e42.slice",
		"io.kubernetes.cri-o.ContainerID": "d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0",
		"io.kubernetes.cri-o.ContainerName": "k8s_POD_clamav-t95hr_clamav_682811c3-f42c-11e8-8dc8-0226acc80e42_6",
		"io.kubernetes.cri-o.ContainerType": "sandbox",
		"io.kubernetes.cri-o.Created": "2019-03-13T12:47:39.635802754Z",
		"io.kubernetes.cri-o.HostName": "clamav-t95hr",
		"io.kubernetes.cri-o.HostNetwork": "false",
		"io.kubernetes.cri-o.HostnamePath": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/hostname",
		"io.kubernetes.cri-o.KubeName": "clamav-t95hr",
		"io.kubernetes.cri-o.Labels": "{\"app\":\"clamav\",\"controller-revision-hash\":\"1279576357\",\"io.kubernetes.container.name\":\"POD\",\"io.kubernetes.pod.name\":\"clamav-t95hr\",\"io.kubernetes.pod.namespace\":\"clamav\",\"io.kubernetes.pod.uid\":\"682811c3-f42c-11e8-8dc8-0226acc80e42\",\"name\":\"clamav\",\"pod-template-generation\":\"1\",\"role\":\"security\"}",
		"io.kubernetes.cri-o.LogPath": "/var/log/pods/682811c3-f42c-11e8-8dc8-0226acc80e42/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0.log",
		"io.kubernetes.cri-o.Metadata": "{\"name\":\"clamav-t95hr\",\"uid\":\"682811c3-f42c-11e8-8dc8-0226acc80e42\",\"namespace\":\"clamav\",\"attempt\":6}",
		"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/92cbed9223a085575a9f3cf7e5dcf6ea7215e8f8f89d6a7f8a1246a0ed4e6523/merged",
		"io.kubernetes.cri-o.Name": "k8s_clamav-t95hr_clamav_682811c3-f42c-11e8-8dc8-0226acc80e42_6",
		"io.kubernetes.cri-o.Namespace": "clamav",
		"io.kubernetes.cri-o.NamespaceOptions": "{\"pid\":1}",
		"io.kubernetes.cri-o.PortMappings": "null",
		"io.kubernetes.cri-o.PrivilegedRuntime": "true",
		"io.kubernetes.cri-o.ResolvPath": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/resolv.conf",
		"io.kubernetes.cri-o.SandboxID": "d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0",
		"io.kubernetes.cri-o.SeccompProfilePath": "",
		"io.kubernetes.cri-o.ShmPath": "/var/run/containers/storage/overlay-containers/d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0/userdata/shm",
		"io.kubernetes.cri-o.TrustedSandbox": "true",
		"io.kubernetes.pod.name": "clamav-t95hr",
		"io.kubernetes.pod.namespace": "clamav",
		"io.kubernetes.pod.uid": "682811c3-f42c-11e8-8dc8-0226acc80e42",
		"kubernetes.io/config.seen": "2019-02-28T16:54:30.880812155Z",
		"kubernetes.io/config.source": "api",
		"name": "clamav",
		"pod-template-generation": "1",
		"role": "security"
	},
	"linux": {
		"resources": {
			"devices": [
				{
					"allow": false,
					"access": "rwm"
				}
			],
			"cpu": {
				"shares": 2
			}
		},
		"cgroupsPath": "kubepods-pod682811c3_f42c_11e8_8dc8_0226acc80e42.slice:crio:d00d308e5286c3f99ac041aa720b901379d9e747a66961e295b634e6cc3237b0",
		"namespaces": [
			{
				"type": "pid"
			},
			{
				"type": "network"
			},
			{
				"type": "ipc"
			},
			{
				"type": "uts"
			},
			{
				"type": "mount"
			}
		],
		"seccomp": {
			"defaultAction": "SCMP_ACT_ERRNO",
			"architectures": [
				"SCMP_ARCH_X86_64",
				"SCMP_ARCH_X86",
				"SCMP_ARCH_X32"
			],
			"syscalls": [
				{
					"names": [
						"accept",
						"accept4",
						"access",
						"alarm",
						"bind",
						"brk",
						"capget",
						"capset",
						"chdir",
						"chmod",
						"chown",
						"chown32",
						"clock_getres",
						"clock_gettime",
						"clock_nanosleep",
						"close",
						"connect",
						"copy_file_range",
						"creat",
						"dup",
						"dup2",
						"dup3",
						"epoll_create",
						"epoll_create1",
						"epoll_ctl",
						"epoll_ctl_old",
						"epoll_pwait",
						"epoll_wait",
						"epoll_wait_old",
						"eventfd",
						"eventfd2",
						"execve",
						"execveat",
						"exit",
						"exit_group",
						"faccessat",
						"fadvise64",
						"fadvise64_64",
						"fallocate",
						"fanotify_mark",
						"fchdir",
						"fchmod",
						"fchmodat",
						"fchown",
						"fchown32",
						"fchownat",
						"fcntl",
						"fcntl64",
						"fdatasync",
						"fgetxattr",
						"flistxattr",
						"flock",
						"fork",
						"fremovexattr",
						"fsetxattr",
						"fstat",
						"fstat64",
						"fstatat64",
						"fstatfs",
						"fstatfs64",
						"fsync",
						"ftruncate",
						"ftruncate64",
						"futex",
						"futimesat",
						"getcpu",
						"getcwd",
						"getdents",
						"getdents64",
						"getegid",
						"getegid32",
						"geteuid",
						"geteuid32",
						"getgid",
						"getgid32",
						"getgroups",
						"getgroups32",
						"getitimer",
						"getpeername",
						"getpgid",
						"getpgrp",
						"getpid",
						"getppid",
						"getpriority",
						"getrandom",
						"getresgid",
						"getresgid32",
						"getresuid",
						"getresuid32",
						"getrlimit",
						"get_robust_list",
						"getrusage",
						"getsid",
						"getsockname",
						"getsockopt",
						"get_thread_area",
						"gettid",
						"gettimeofday",
						"getuid",
						"getuid32",
						"getxattr",
						"inotify_add_watch",
						"inotify_init",
						"inotify_init1",
						"inotify_rm_watch",
						"io_cancel",
						"ioctl",
						"io_destroy",
						"io_getevents",
						"ioprio_get",
						"ioprio_set",
						"io_setup",
						"io_submit",
						"ipc",
						"kill",
						"lchown",
						"lchown32",
						"lgetxattr",
						"link",
						"linkat",
						"listen",
						"listxattr",
						"llistxattr",
						"_llseek",
						"lremovexattr",
						"lseek",
						"lsetxattr",
						"lstat",
						"lstat64",
						"madvise",
						"memfd_create",
						"mincore",
						"mkdir",
						"mkdirat",
						"mknod",
						"mknodat",
						"mlock",
						"mlock2",
						"mlockall",
						"mmap",
						"mmap2",
						"mprotect",
						"mq_getsetattr",
						"mq_notify",
						"mq_open",
						"mq_timedreceive",
						"mq_timedsend",
						"mq_unlink",
						"mremap",
						"msgctl",
						"msgget",
						"msgrcv",
						"msgsnd",
						"msync",
						"munlock",
						"munlockall",
						"munmap",
						"nanosleep",
						"newfstatat",
						"_newselect",
						"open",
						"openat",
						"pause",
						"pipe",
						"pipe2",
						"poll",
						"ppoll",
						"prctl",
						"pread64",
						"preadv",
						"prlimit64",
						"pselect6",
						"pwrite64",
						"pwritev",
						"read",
						"readahead",
						"readlink",
						"readlinkat",
						"readv",
						"recv",
						"recvfrom",
						"recvmmsg",
						"recvmsg",
						"remap_file_pages",
						"removexattr",
						"rename",
						"renameat",
						"renameat2",
						"restart_syscall",
						"rmdir",
						"rt_sigaction",
						"rt_sigpending",
						"rt_sigprocmask",
						"rt_sigqueueinfo",
						"rt_sigreturn",
						"rt_sigsuspend",
						"rt_sigtimedwait",
						"rt_tgsigqueueinfo",
						"sched_getaffinity",
						"sched_getattr",
						"sched_getparam",
						"sched_get_priority_max",
						"sched_get_priority_min",
						"sched_getscheduler",
						"sched_rr_get_interval",
						"sched_setaffinity",
						"sched_setattr",
						"sched_setparam",
						"sched_setscheduler",
						"sched_yield",
						"seccomp",
						"select",
						"semctl",
						"semget",
						"semop",
						"semtimedop",
						"send",
						"sendfile",
						"sendfile64",
						"sendmmsg",
						"sendmsg",
						"sendto",
						"setfsgid",
						"setfsgid32",
						"setfsuid",
						"setfsuid32",
						"setgid",
						"setgid32",
						"setgroups",
						"setgroups32",
						"setitimer",
						"setpgid",
						"setpriority",
						"setregid",
						"setregid32",
						"setresgid",
						"setresgid32",
						"setresuid",
						"setresuid32",
						"setreuid",
						"setreuid32",
						"setrlimit",
						"set_robust_list",
						"setsid",
						"setsockopt",
						"set_thread_area",
						"set_tid_address",
						"setuid",
						"setuid32",
						"setxattr",
						"shmat",
						"shmctl",
						"shmdt",
						"shmget",
						"shutdown",
						"sigaltstack",
						"signalfd",
						"signalfd4",
						"sigreturn",
						"socket",
						"socketcall",
						"socketpair",
						"splice",
						"stat",
						"stat64",
						"statfs",
						"statfs64",
						"symlink",
						"symlinkat",
						"sync",
						"sync_file_range",
						"syncfs",
						"sysinfo",
						"syslog",
						"tee",
						"tgkill",
						"time",
						"timer_create",
						"timer_delete",
						"timerfd_create",
						"timerfd_gettime",
						"timerfd_settime",
						"timer_getoverrun",
						"timer_gettime",
						"timer_settime",
						"times",
						"tkill",
						"truncate",
						"truncate64",
						"ugetrlimit",
						"umask",
						"uname",
						"unlink",
						"unlinkat",
						"utime",
						"utimensat",
						"utimes",
						"vfork",
						"vmsplice",
						"wait4",
						"waitid",
						"waitpid",
						"write",
						"writev"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"personality"
					],
					"action": "SCMP_ACT_ALLOW",
					"args": [
						{
							"index": 0,
							"value": 0,
							"op": "SCMP_CMP_EQ"
						},
						{
							"index": 0,
							"value": 8,
							"op": "SCMP_CMP_EQ"
						},
						{
							"index": 0,
							"value": 4294967295,
							"op": "SCMP_CMP_EQ"
						}
					]
				},
				{
					"names": [
						"chroot"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"clone"
					],
					"action": "SCMP_ACT_ALLOW",
					"args": [
						{
							"index": 0,
							"value": 2080505856,
							"op": "SCMP_CMP_MASKED_EQ"
						}
					]
				},
				{
					"names": [
						"arch_prctl"
					],
					"action": "SCMP_ACT_ALLOW"
				},
				{
					"names": [
						"modify_ldt"
					],
					"action": "SCMP_ACT_ALLOW"
				}
			]
		},
		"mountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c45,c386"
	}
}
@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 15, 2019

@bhperry Could you verify if both of these processes uts namespaces are same or different from /proc/1/ns/uts?

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 15, 2019

What I see so far is that pod has requested a uts namespace different from the host and if it still matches the host then something else is at play.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 15, 2019

They are the same.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 15, 2019

@bhperry so they match the host uts, right?
@giuseppe This probably points to a runc issue which is puzzling.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 16, 2019

@bhperry Can you share the clamav daemonset yaml?

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 16, 2019

Clamav container has the same UTS namespace as the host. Does the pod have a UTS namespace as well? I don't know how to find that if it does.

Here's the daemonset yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: clamav
  namespace: clamav
  labels:
    name: clamav-daemonset
    app: clamav
spec:
  selector:
    matchLabels:
      name: clamav
      app:  clamav
      role: security
  template:
    metadata:
      labels:
        name: clamav
        app: clamav
        role: security
    spec:
      serviceAccount: clamav-account
      containers:
        - name: clamav
          image: docker.io/datica/clamav:latest
          securityContext:
            privileged: true
          resources:
            limits:
              memory: 1Gi
              cpu: 0.5
          volumeMounts:
            - mountPath: /host
              name: host
              readOnly: true
            - mountPath: /etc/clamav
              name: clamav-config
            - mountPath: /var/log/clamav
              name: clamav-logs
            - mountPath: /root/.aws
              name: aws-config
      tolerations:
      - key: node-role.kubernetes.io/controller
        operator: Exists
        effect: NoSchedule
      volumes:
        - name: host
          hostPath:
            path: /
        - name: clamav-config
          configMap:
            name: clamav-config
        - name: clamav-logs
          hostPath:
            path: /var/log/clamav
        - name: aws-config
          secret:
            secretName: aws-config-for-clamav
@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 16, 2019

@bhperry, you can do the same to find the pod uts namespace.

runc list | grep pod id
ls -l /proc/pod_pid/ns/uts

From the data you have provided so far it looks like the pod is requesting a uts namespace which is separate from the host and the container simply joins it i.e. uses the pods uts namespace so I expect that you will see the same uts namespace for the pod but it will be good to double check.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 18, 2019

Yes, pod is the same as the host and container.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 19, 2019

@bhperry, this only happens with clamav daemon set for you? If there are any other daemonsets that are seeing this issue then we can try to find a pattern. Thing that stands out so far is mounting of /. I will attempt to recreate this locally.

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 20, 2019

For the most part, yes. However there is actually one case where this happened with an ingress-nginx default-http-backend pod. It has a similar event relating to the IPC namespace like what I've seen when clamav triggers the bug and is stuck in CreateContainerError

      Warning  Failed  6m (x96640 over 27d)  kubelet, ip-10-240-4-42.us-west-2.compute.internal  Error: container create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1713: running lstat on namespace path \"/proc/18926/ns/ipc\" caused \"lstat /proc/18926/ns/ipc: no such file or directory\""

Here is the yaml for the replicaset that controls it. It somehow got orphaned from its deployment:

    apiVersion: extensions/v1beta1
    kind: ReplicaSet
    metadata:
      annotations:
        deployment.kubernetes.io/desired-replicas: "1"
        deployment.kubernetes.io/max-replicas: "2"
        deployment.kubernetes.io/revision: "1"
      creationTimestamp: 2018-08-30T21:53:32Z
      generation: 1
      labels:
        app.kubernetes.io/name: default-http-backend
        app.kubernetes.io/part-of: ingress-nginx
        pod-template-hash: "315190807"
      name: default-http-backend-7595f4d4c
      namespace: ingress-nginx
      resourceVersion: "38705563"
      selfLink: /apis/extensions/v1beta1/namespaces/ingress-nginx/replicasets/default-http-backend-7595f4d4c
      uid: 22f17178-ac9f-11e8-9eb8-0a5a0031a31c
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: default-http-backend
          pod-template-hash: "315190807"
      template:
        metadata:
          creationTimestamp: null
          labels:
            app.kubernetes.io/name: default-http-backend
            app.kubernetes.io/part-of: ingress-nginx
            pod-template-hash: "315190807"
        spec:
          containers:
          - image: gcr.io/google_containers/defaultbackend:1.4
            imagePullPolicy: IfNotPresent
            livenessProbe:
              failureThreshold: 3
              httpGet:
                path: /healthz
                port: 8080
                scheme: HTTP
              initialDelaySeconds: 30
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 5
            name: default-http-backend
            ports:
            - containerPort: 8080
              protocol: TCP
            resources:
              limits:
                cpu: 10m
                memory: 20Mi
              requests:
                cpu: 10m
                memory: 20Mi
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 60
    status:
      fullyLabeledReplicas: 1
      observedGeneration: 1
      replicas: 1

Unfortunately, since the pod and container are not running I am unable to verify if the UTS namespace matches the host or not.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 20, 2019

@bhperry That is very useful to know. One PR that @giuseppe has opened to address the lstat issues is here - kubernetes/kubernetes#72105. It is a potential race between kubelet and runtime. @giuseppe Maybe we can run your stress tests with the clamav daemonset/pod and see if we can hit the uts namespace being shared with host.

@mrunalp

This comment has been minimized.

Copy link
Member

@mrunalp mrunalp commented Mar 20, 2019

I have opened a backport #2143 which should help with the namespace join issue. We will cut a new release once it is merged.

@giuseppe

This comment has been minimized.

Copy link
Member

@giuseppe giuseppe commented Mar 20, 2019

with daemon sets it is quite easy to reproduce that error, create a daemonset. The CRI-O patches fix partially the issue, but I was still seeing it without the fix for Kubernetes.

Do you bind mount / recursively? I am not sure /host/proc should be inside of the container, it could still be possible for a program inside of the container to join a different namespace (such as host UTS) if /proc is available

@bhperry

This comment has been minimized.

Copy link

@bhperry bhperry commented Mar 20, 2019

Great to hear there's some fixes for some of this, looking forward to getting those updates.

I'm not actually sure why we have / bound, can't remember what the reasoning was for that. What do you mean by binding it recursively?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.