Failed to get D-Bus connection: Operation not permitted #1399

free2k · 2019-03-21T07:00:31Z

Description of problem

➜ ~ docker run --runtime kata-runtime -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name kata-test --cpus 2 --memory 2G -itd crawler:v2
8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33
➜ ~ docker exec -it 8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 /bin/sh
sh-4.2# systemctl
Failed to get D-Bus connection: Operation not permitted

Expected result

Can successfully run systemd

Actual result

Failed to get D-Bus connection: Operation not permitted

Meta details

Running kata-collect-data.sh version 1.6.0-rc1 (commit 9f8d4e1) at 2019-03-21.15:00:00.853126502+0800.

Runtime is /usr/bin/kata-runtime.

`kata-env`

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.6.0-rc1"
    Commit = "9f8d4e1"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.6.0-rc1_agent_a2037c08531.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.19.24.25-20.1.container"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.6.0-rc1-2085829"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.6.0-rc1-0f41347"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "3.10.0-693.mt20180601.62.el7.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "CentOS Linux"
    Version = "7"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz"

[Netmon]
  Version = "kata-netmon version 1.6.0-rc1"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found
Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"

# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true

# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true

# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
#
#   - none
#     Used when customize network. Only creates a tap device. No veth pair.
#
#   - tcfilter
#     Uses tc filter rules to redirect traffic from the network interface
#     provided by plugin to a tap interface connected to the VM.
#
internetworking_model="macvtap"

# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true

KSM throttler

version

Output of "/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":

kata-ksm-throttler version 1.6.0-rc1-ce5b765

Output of "/usr/lib/systemd/system/kata-ksm-throttler.service --version":

/usr/bin/kata-collect-data.sh:行168: /usr/lib/systemd/system/kata-ksm-throttler.service: 权限不够

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2019-03-04T21:52:40.031090715+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "28100"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.6.0-rc1-a2037c08531e4982003d8fc9b4837d27c35c0966"
  agent-is-init-daemon: "no"

Initrd details

No initrd

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2019-03-15T16:51:10.033879262+08:00" level=error msg="QMP command failed" arch=amd64 command=create container=931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b name=kata-runtime pid=13891 source=runtime
time="2019-03-15T16:51:10.064978884+08:00" level=error msg="Container ID (931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b) does not exist" arch=amd64 command=delete container=931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b name=kata-runtime pid=13955 source=runtime
time="2019-03-15T16:51:10.091568255+08:00" level=error msg="Container ID (931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b) does not exist" arch=amd64 command=delete container=931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b name=kata-runtime pid=13964 source=runtime
time="2019-03-15T16:52:39.575263368+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b name=kata-runtime pid=14132 source=virtcontainers subsystem=kata_agent
time="2019-03-15T16:52:39.575292176+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b error="open /run/vc/sbs/fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b/devices.json: no such file or directory" name=kata-runtime pid=14132 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b sandboxid=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=virtcontainers subsystem=sandbox
time="2019-03-15T16:52:39.577442065+08:00" level=debug arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=14132 source=virtcontainers subsystem=qemu
time="2019-03-15T16:52:40.30279985+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b name=kata-runtime pid=14132 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-03-15T16:52:40.302893474+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b destination="fe80::/64" name=kata-runtime pid=14132 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-03-15T16:52:40.461009097+08:00" level=error msg="rollback failed removeResources()" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b error="Unable to remove 2 CPUs, currently there are only 0 hotplugged CPUs" name=kata-runtime pid=14132 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=virtcontainers subsystem=container
time="2019-03-15T16:52:40.529880603+08:00" level=error msg="QMP command failed" arch=amd64 command=create container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b name=kata-runtime pid=14132 source=runtime
time="2019-03-15T16:52:40.557981422+08:00" level=error msg="Container ID (fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b) does not exist" arch=amd64 command=delete container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b name=kata-runtime pid=14196 source=runtime
time="2019-03-15T16:52:40.591456024+08:00" level=error msg="Container ID (fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b) does not exist" arch=amd64 command=delete container=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b name=kata-runtime pid=14206 source=runtime
time="2019-03-15T16:53:32.104577601+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e name=kata-runtime pid=14343 source=virtcontainers subsystem=kata_agent
time="2019-03-15T16:53:32.104606373+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e error="open /run/vc/sbs/59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e/devices.json: no such file or directory" name=kata-runtime pid=14343 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e sandboxid=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=virtcontainers subsystem=sandbox
time="2019-03-15T16:53:32.106915006+08:00" level=debug arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=14343 source=virtcontainers subsystem=qemu
time="2019-03-15T16:53:32.826008239+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e name=kata-runtime pid=14343 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-03-15T16:53:32.826095555+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e destination="fe80::/64" name=kata-runtime pid=14343 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-03-15T16:53:33.011935563+08:00" level=error msg="rollback failed removeResources()" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e error="Unable to remove 2 CPUs, currently there are only 0 hotplugged CPUs" name=kata-runtime pid=14343 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=virtcontainers subsystem=container
time="2019-03-15T16:53:33.075879758+08:00" level=error msg="QMP command failed" arch=amd64 command=create container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e name=kata-runtime pid=14343 source=runtime
time="2019-03-15T16:53:33.102985855+08:00" level=error msg="Container ID (59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e) does not exist" arch=amd64 command=delete container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e name=kata-runtime pid=14407 source=runtime
time="2019-03-15T16:53:33.126871092+08:00" level=error msg="Container ID (59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e) does not exist" arch=amd64 command=delete container=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e name=kata-runtime pid=14416 source=runtime
time="2019-03-15T18:37:54.728363858+08:00" level=error msg="hook error" arch=amd64 command=create container=8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e error="exit status 1: stdout: , stderr: " hook-type=pre-start name=kata-runtime pid=3667 source=katautils subsystem=hook
time="2019-03-15T18:37:54.72843999+08:00" level=error msg="exit status 1: stdout: , stderr: " arch=amd64 command=create container=8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e name=kata-runtime pid=3667 source=runtime
time="2019-03-15T18:37:54.74835566+08:00" level=error msg="Container ID (8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e) does not exist" arch=amd64 command=delete container=8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e name=kata-runtime pid=3700 source=runtime
time="2019-03-15T18:37:54.788570632+08:00" level=error msg="Container ID (8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e) does not exist" arch=amd64 command=delete container=8d9449b10cb045634c37d6052e64787e9b7ff7e83a94424dd22d050e8b56055e name=kata-runtime pid=3715 source=runtime
time="2019-03-15T18:41:00.41049659+08:00" level=error msg="hook error" arch=amd64 command=create container=744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40 error="exit status 1: stdout: , stderr: " hook-type=pre-start name=kata-runtime pid=3921 source=katautils subsystem=hook
time="2019-03-15T18:41:00.410556207+08:00" level=error msg="exit status 1: stdout: , stderr: " arch=amd64 command=create container=744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40 name=kata-runtime pid=3921 source=runtime
time="2019-03-15T18:41:00.430803518+08:00" level=error msg="Container ID (744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40) does not exist" arch=amd64 command=delete container=744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40 name=kata-runtime pid=3951 source=runtime
time="2019-03-15T18:41:00.480079326+08:00" level=error msg="Container ID (744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40) does not exist" arch=amd64 command=delete container=744c409d2bcafa0152e4f512456aa50fba0de7b102c7f53bc2bfcc48cb969c40 name=kata-runtime pid=3962 source=runtime
time="2019-03-15T18:44:17.344894774+08:00" level=error msg="hook error" arch=amd64 command=create container=e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454 error="exit status 1: stdout: , stderr: " hook-type=pre-start name=kata-runtime pid=4230 source=katautils subsystem=hook
time="2019-03-15T18:44:17.344958762+08:00" level=error msg="exit status 1: stdout: , stderr: " arch=amd64 command=create container=e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454 name=kata-runtime pid=4230 source=runtime
time="2019-03-15T18:44:17.365138642+08:00" level=error msg="Container ID (e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454) does not exist" arch=amd64 command=delete container=e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454 name=kata-runtime pid=4259 source=runtime
time="2019-03-15T18:44:17.40111825+08:00" level=error msg="Container ID (e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454) does not exist" arch=amd64 command=delete container=e08e00fb4e86d011519b0df279d1d353cfddc9142017403c32b8119a7d133454 name=kata-runtime pid=4270 source=runtime
time="2019-03-15T18:44:54.225486159+08:00" level=error msg="hook error" arch=amd64 command=create container=b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a error="exit status 1: stdout: , stderr: " hook-type=pre-start name=kata-runtime pid=4438 source=katautils subsystem=hook
time="2019-03-15T18:44:54.225546553+08:00" level=error msg="exit status 1: stdout: , stderr: " arch=amd64 command=create container=b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a name=kata-runtime pid=4438 source=runtime
time="2019-03-15T18:44:54.245780565+08:00" level=error msg="Container ID (b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a) does not exist" arch=amd64 command=delete container=b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a name=kata-runtime pid=4467 source=runtime
time="2019-03-15T18:44:54.281370701+08:00" level=error msg="Container ID (b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a) does not exist" arch=amd64 command=delete container=b33f6d527c25683c56bcbc4dc1b91b1806267efb7cd0344d3c5052400d74e90a name=kata-runtime pid=4478 source=runtime
time="2019-03-15T18:45:56.807627887+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 error="open /run/vc/sbs/8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513/devices.json: no such file or directory" name=kata-runtime pid=4678 sandbox=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 sandboxid=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 source=virtcontainers subsystem=sandbox
time="2019-03-15T18:45:58.548430776+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 name=kata-runtime pid=4678 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-03-15T18:45:58.548523684+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 destination="fe80::/64" name=kata-runtime pid=4678 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-03-21T14:55:44.200153736+08:00" level=error msg="Container ID (8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513) does not exist" arch=amd64 command=delete container=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 name=kata-runtime pid=27675 source=runtime
time="2019-03-21T14:56:02.60224772+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b error="open /run/vc/sbs/dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b/devices.json: no such file or directory" name=kata-runtime pid=27865 sandbox=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b sandboxid=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b source=virtcontainers subsystem=sandbox
time="2019-03-21T14:56:03.224121283+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b name=kata-runtime pid=27865 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-03-21T14:56:03.224207262+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b destination="fe80::/64" name=kata-runtime pid=27865 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-03-21T14:58:01.071396951+08:00" level=error msg="unknown FS magic on \"/var/run/netns/cni-178aab79-a80e-84bc-8b38-b5c2dd9b27b4\": 1021994" arch=amd64 command=kill container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b name=kata-runtime pid=28104 sandbox=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b source=runtime
time="2019-03-21T14:58:01.25659285+08:00" level=error msg="cgroups: unable to remove paths /sys/fs/cgroup/memory/kata/system.slice:docker:dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b" arch=amd64 command=delete container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b name=kata-runtime pid=28127 sandbox=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b source=runtime
time="2019-03-21T14:58:01.283030662+08:00" level=error msg="Container not ready or stopped, impossible to delete" arch=amd64 command=delete container=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b name=kata-runtime pid=28141 sandbox=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b source=runtime
time="2019-03-21T14:58:10.842227646+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 error="open /run/vc/sbs/8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33/devices.json: no such file or directory" name=kata-runtime pid=28262 sandbox=8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 sandboxid=8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 source=virtcontainers subsystem=sandbox
time="2019-03-21T14:58:11.463125979+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 name=kata-runtime pid=28262 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-03-21T14:58:11.463211973+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=8838e723bd0cfd53b99886d415aef74fcf8de177b0ac8c62cabeec261dfa3f33 destination="fe80::/64" name=kata-runtime pid=28262 source=virtcontainers subsystem=network unsupported-route-type=ipv6

Proxy logs

Recent proxy problems found in system journal:

time="2019-03-15T16:07:11.943588065+08:00" level=info msg="[  424.045468] run queue from wrong CPU 2, hctx active\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:07:11.944060101+08:00" level=info msg="[  424.045936] run queue from wrong CPU 2, hctx active\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:07:11.99572204+08:00" level=info msg="[  424.097601] run queue from wrong CPU 2, hctx active\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:46:07.765891856+08:00" level=info msg="time=\"2019-03-15T08:46:07.724154769Z\" level=debug msg=\"request end\" duration=\"459.965µs\" name=kata-agent pid=109 request=/grpc.AgentService/ReadStdout resp=\"data:\\\"Failed to get D-Bus connection: Operation not permitted\\\\r\\\\n\\\" \" sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:01.483533463+08:00" level=info msg="time=\"2019-03-15T08:48:01.441581502Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=109 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 signal=\"child exited\" source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:01.483920964+08:00" level=info msg="time=\"2019-03-15T08:48:01.44197338Z\" level=debug msg=\"request end\" duration=\"14.673µs\" error=\"read /dev/ptmx: input/output error\" name=kata-agent pid=109 request=/grpc.AgentService/ReadStdout resp= sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:15.051061001+08:00" level=info msg="time=\"2019-03-15T08:48:15.009086735Z\" level=debug msg=\"request end\" duration=\"85.706µs\" name=kata-agent pid=109 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.057814549+08:00" level=info msg="time=\"2019-03-15T08:48:25.015737941Z\" level=debug msg=\"request end\" duration=\"376.137µs\" name=kata-agent pid=109 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.05891072+08:00" level=info msg="time=\"2019-03-15T08:48:25.016269722Z\" level=debug msg=\"request end\" duration=48m16.541147754s error=\"read /dev/ptmx: input/output error\" name=kata-agent pid=109 request=/grpc.AgentService/ReadStdout resp= sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.059438989+08:00" level=info msg="time=\"2019-03-15T08:48:25.017450814Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=109 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 signal=\"child exited\" source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.180381683+08:00" level=info msg="[ 2897.277189] run queue from wrong CPU 2, hctx active\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.254369268+08:00" level=info msg="[ 2897.351173] run queue from wrong CPU 2, hctx active\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.321609214+08:00" level=info msg="time=\"2019-03-15T08:48:25.279608989Z\" level=debug msg=\"request end\" duration=260.957153ms name=kata-agent pid=109 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{}\" sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.376605897+08:00" level=info msg="[ 2897.473392] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.391438713+08:00" level=info msg="time=\"2019-03-15T08:48:25.349436228Z\" level=debug msg=\"request end\" duration=2.165347ms name=kata-agent pid=109 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{}\" sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent\n" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=agent
time="2019-03-15T16:48:25.391670144+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318/proxy.sock: use of closed network connection" name=kata-proxy pid=12394 sandbox=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 source=proxy
time="2019-03-15T16:50:06.651759741+08:00" level=info msg="[    0.213724] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=13664 sandbox=88e3c4d5f657362e2297f69bc58feb273eb32e1c73b13e5f2ea78935fc546a0c source=agent
time="2019-03-15T16:50:06.652046272+08:00" level=info msg="[    0.213987] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=13664 sandbox=88e3c4d5f657362e2297f69bc58feb273eb32e1c73b13e5f2ea78935fc546a0c source=agent
time="2019-03-15T16:50:07.042251664+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/88e3c4d5f657362e2297f69bc58feb273eb32e1c73b13e5f2ea78935fc546a0c/kata.sock: use of closed network connection" name=kata-proxy pid=13664 sandbox=88e3c4d5f657362e2297f69bc58feb273eb32e1c73b13e5f2ea78935fc546a0c source=proxy
time="2019-03-15T16:51:10.010534968+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b/kata.sock: use of closed network connection" name=kata-proxy pid=13933 sandbox=931303b4d307a63c8e128b41097c482dcf50076bc43feb4e8c33f8c2d22d6f6b source=proxy
time="2019-03-15T16:52:40.117574649+08:00" level=info msg="[    0.217087] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.117865061+08:00" level=info msg="[    0.217354] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.141771067+08:00" level=info msg="[    0.241260] systemd-gpt-auto-generator[67]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.16588583+08:00" level=info msg="[    0.265359] systemd-journald[71]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.180147999+08:00" level=info msg="[    0.279658] systemd[75]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.19471465+08:00" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.195507826+08:00" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.309529947+08:00" level=debug msg="Copy stream error" error="write unix /run/vc/sbs/fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b/proxy.sock->@: write: broken pipe" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=proxy
time="2019-03-15T16:52:40.315622338+08:00" level=info msg="time=\"2019-03-15T08:52:40.28046949Z\" level=debug msg=\"request end\" duration=3.753667ms name=kata-agent pid=108 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{}\" sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.360465842+08:00" level=info msg="[    0.459955] sd 0:0:0:6: [sdg] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.36107832+08:00" level=info msg="[    0.460577] sd 0:0:0:6: [sdg] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.361927221+08:00" level=info msg="[    0.461422] sd 0:0:0:6: [sdg] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.363395717+08:00" level=info msg="[    0.462893] sd 0:0:0:6: [sdg] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=agent
time="2019-03-15T16:52:40.506807506+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b/kata.sock: use of closed network connection" name=kata-proxy pid=14170 sandbox=fda3986c71db7a10479586383fa984343884d6a4b1d04b7cc7e443fee446430b source=proxy
time="2019-03-15T16:53:32.645380395+08:00" level=info msg="[    0.213683] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.64568323+08:00" level=info msg="[    0.213959] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.668162658+08:00" level=info msg="[    0.236458] systemd-gpt-auto-generator[67]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.687863049+08:00" level=info msg="[    0.256151] systemd-journald[69]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.694013208+08:00" level=info msg="[    0.262323] systemd[71]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.719235743+08:00" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.720661542+08:00" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.840036803+08:00" level=info msg="time=\"2019-03-15T08:53:32.803556987Z\" level=debug msg=\"request end\" duration=7.263178ms name=kata-agent pid=108 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{}\" sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.889355557+08:00" level=info msg="[    0.457651] sd 0:0:0:6: [sdg] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.889602785+08:00" level=info msg="[    0.457896] sd 0:0:0:6: [sdg] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.891565775+08:00" level=info msg="[    0.459861] sd 0:0:0:6: [sdg] Read Capacity(16) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:32.892608301+08:00" level=info msg="[    0.460752] sd 0:0:0:6: [sdg] Read Capacity(10) failed: Result: hostbyte=0x00 driverbyte=0x08\n" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=agent
time="2019-03-15T16:53:33.055070849+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e/proxy.sock: use of closed network connection" name=kata-proxy pid=14384 sandbox=59f498aa25a2b235f2e4c8a89630dd506b7e2ba3d9c9bc1d86e435671d57140e source=proxy
time="2019-03-15T17:03:02.596964291+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/7b051c5658e9f1738bfabec4a702e179595c667a91ee5e06ba3945f1ef55a539/proxy.sock: use of closed network connection" name=kata-proxy pid=9030 sandbox=7b051c5658e9f1738bfabec4a702e179595c667a91ee5e06ba3945f1ef55a539 source=proxy
time="2019-03-21T14:55:44.125003795+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513/kata.sock: use of closed network connection" name=kata-proxy pid=4728 sandbox=8370fcd81b51641effc53f1a2464246eb446f01034d92540b32b13738f9a5513 source=proxy
time="2019-03-21T14:58:01.070252444+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b/kata.sock: use of closed network connection" name=kata-proxy pid=27905 sandbox=dc54adc9cb5ddf514f80076babc8f648ef6c703a3178b43ed554865ad9e18b2b source=proxy

Shim logs

Recent shim problems found in system journal:

time="2019-03-07T17:28:36.663128453+08:00" level=info msg="copy stdout failed" container=c3d6d0c8ba3c5750ba890342e2456a38c4da34d101d04d676a56c1e244c7a33d error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=f8f866dc-ee7f-46b8-b97f-2f5afa67d899 name=kata-shim pid=11 source=shim
time="2019-03-07T17:44:25.074943428+08:00" level=info msg="copy stdout failed" container=cea06cc5f93a00486008def51edb5d3e6507fb2a20adf52566a52d10105f7e15 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=311e5458-bb54-484d-b756-7f7324a4dfc7 name=kata-shim pid=11 source=shim
time="2019-03-07T17:52:53.347436355+08:00" level=info msg="copy stdout failed" container=8d5f6cea794ff63e2b583974f922d86fd61d2a6b2e9976b9b1115f010e7c0c06 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=b6617fd3-89bf-4f65-9780-f9a0d0aa8c50 name=kata-shim pid=11 source=shim
time="2019-03-07T17:59:44.398780296+08:00" level=info msg="copy stdout failed" container=eb8e81d486f3b80d7c147dc2b78633709c43563e8c51a46ec0f2fbe59db92846 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=6a5ba2bc-226e-4227-8066-e224484c88de name=kata-shim pid=11 source=shim
time="2019-03-07T18:00:44.739603387+08:00" level=info msg="copy stdout failed" container=7327b6ad494f053d4c2e589a43240c1dbba85adce640ad59705395becb32b21a error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=7327b6ad494f053d4c2e589a43240c1dbba85adce640ad59705395becb32b21a name=kata-shim pid=1 source=shim
time="2019-03-07T19:13:17.900259357+08:00" level=info msg="copy stdout failed" container=5b95ed917f1d9f5e36f2431bb528185e9496055d61a8f00a51e0937b53509f15 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=18bbe9c8-959f-4749-8269-109c735e21ed name=kata-shim pid=12 source=shim
time="2019-03-07T20:15:03.348398656+08:00" level=info msg="copy stdout failed" container=721d95d21d97fa254735dac40c0db86c5e93b538344c12397f50abbfb2060207 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=93879a3a-f5dc-4772-b097-afcee1bb0256 name=kata-shim pid=21 source=shim
time="2019-03-07T20:41:00.952367111+08:00" level=info msg="copy stdout failed" container=c4c96fe27df8d27ad342fa5d6ed4201fc91ad7338573471971e770b9d0841842 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=03bba5f6-c2af-4c25-99e4-dff39d477b94 name=kata-shim pid=12 source=shim
time="2019-03-07T20:51:59.403201028+08:00" level=info msg="copy stdout failed" container=eb199cfb1510c25ca43c013a2a099d12e738c7aaf0e3186db96ab9ba13559a61 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=70efc524-a2d5-4e8f-98c8-3f618b311c6c name=kata-shim pid=11 source=shim
time="2019-03-07T20:52:08.296189125+08:00" level=info msg="copy stdout failed" container=eb199cfb1510c25ca43c013a2a099d12e738c7aaf0e3186db96ab9ba13559a61 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=f37ffe0d-a0b8-4401-a595-3845816db9eb name=kata-shim pid=21 source=shim
time="2019-03-07T20:52:30.394535283+08:00" level=info msg="copy stdout failed" container=eb199cfb1510c25ca43c013a2a099d12e738c7aaf0e3186db96ab9ba13559a61 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=8e1e7087-be58-4335-9ce3-7e8c91e18b41 name=kata-shim pid=31 source=shim
time="2019-03-08T15:29:48.27331984+08:00" level=info msg="copy stdout failed" container=440fb71e6b627809866a01e0df77da05c512ec8c275de11b4c0e897c3cfb9168 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=b39ca589-c964-4547-965b-ace4a636ea0e name=kata-shim pid=11 source=shim
time="2019-03-12T16:52:51.132039001+08:00" level=info msg="copy stdout failed" container=75005e03176a7d3bdcaa284537aaa5599c1c30a932d4e75024bd1ac8d5ef2a30 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=96f7b1b3-0b8a-4907-946f-4c5e76a7e142 name=kata-shim pid=11 source=shim
time="2019-03-12T16:54:51.631602848+08:00" level=info msg="copy stdout failed" container=75005e03176a7d3bdcaa284537aaa5599c1c30a932d4e75024bd1ac8d5ef2a30 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e3099ddb-d598-4964-a18f-4ae0bd7342ab name=kata-shim pid=21 source=shim
time="2019-03-12T16:55:42.109477431+08:00" level=info msg="copy stdout failed" container=75005e03176a7d3bdcaa284537aaa5599c1c30a932d4e75024bd1ac8d5ef2a30 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=75005e03176a7d3bdcaa284537aaa5599c1c30a932d4e75024bd1ac8d5ef2a30 name=kata-shim pid=1 source=shim
time="2019-03-12T17:00:26.597677358+08:00" level=info msg="copy stdout failed" container=5693aa46ef2f264f9aaababd6975a02561ff60dcca2c0561ab91cf1b6e72da6c error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=c91d5c73-f666-41b5-b04b-999a99fb9522 name=kata-shim pid=11 source=shim
time="2019-03-13T14:37:16.126598262+08:00" level=info msg="copy stdout failed" container=7f511a9f5791e546394965f23fa7b542a2164d635eb2f57ecf9ce0569aaa2b28 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=834a8b4e-94a9-47ce-8d7e-baf1b4f916ac name=kata-shim pid=12 source=shim
time="2019-03-13T14:55:51.140937714+08:00" level=info msg="copy stdout failed" container=9920a0f0deb4bb63bc40b86cd68d507f7c0d74c2cb35bc6ce9c863d9d2c4b07e error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=d1bf858f-3841-4c53-b424-133212fd5337 name=kata-shim pid=11 source=shim
time="2019-03-14T17:01:08.507746847+08:00" level=info msg="copy stdout failed" container=e39a1042760e0862a218a03e72843f771081c6a2322d9628571e66b45a69db09 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=3c314491-0efe-468e-be98-0c5f549cfcf6 name=kata-shim pid=12 source=shim
time="2019-03-14T17:06:30.329725893+08:00" level=info msg="copy stdout failed" container=de3ba9058e87ef3e1e5ba906adeb494f1ba7f896b1445a3a0001db970ffa7339 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=7482615c-c1c2-4513-8fae-d9beceae4849 name=kata-shim pid=11 source=shim
time="2019-03-14T17:08:51.683701733+08:00" level=info msg="copy stdout failed" container=f6047a506e68401bf9da4e679da0dc216b4adbb22ef679304cf17eb344e70f4f error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=93f58f98-1123-4296-8125-b806a7e557c5 name=kata-shim pid=11 source=shim
time="2019-03-14T18:28:41.105224446+08:00" level=info msg="copy stdout failed" container=7fb3f63d5ea74c18f2532e28b80a865a20480fc0d3f3631b5a081a7460aa7506 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=7b8e0aee-07d5-4694-9d38-829f3e5fa6df name=kata-shim pid=12 source=shim
time="2019-03-14T18:34:11.854316685+08:00" level=info msg="copy stdout failed" container=7fb3f63d5ea74c18f2532e28b80a865a20480fc0d3f3631b5a081a7460aa7506 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=39e58524-1edf-4f7a-94be-66891bb5312f name=kata-shim pid=22 source=shim
time="2019-03-14T19:58:00.982670614+08:00" level=info msg="copy stdout failed" container=7fb3f63d5ea74c18f2532e28b80a865a20480fc0d3f3631b5a081a7460aa7506 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=9f3ebc05-0470-4e8e-ab54-63fa3488fc68 name=kata-shim pid=31 source=shim
time="2019-03-14T20:19:03.939223791+08:00" level=info msg="copy stdout failed" container=7fb3f63d5ea74c18f2532e28b80a865a20480fc0d3f3631b5a081a7460aa7506 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=697abd9b-9232-42b9-a89b-a10e92cc7f16 name=kata-shim pid=41 source=shim
time="2019-03-14T20:33:35.768308527+08:00" level=info msg="copy stdout failed" container=7fb3f63d5ea74c18f2532e28b80a865a20480fc0d3f3631b5a081a7460aa7506 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=10993d11-33c9-4ea2-981a-fe6bb3ed3af9 name=kata-shim pid=52 source=shim
time="2019-03-15T11:25:29.717898802+08:00" level=info msg="copy stdout failed" container=23316c1c18996b0ed5f642b83c90813571f0617c5e2517366dfb204df304dd77 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=66110349-b796-4acd-be6d-b07fb9d32f20 name=kata-shim pid=24 source=shim
time="2019-03-15T15:05:25.01086428+08:00" level=info msg="copy stdout failed" container=23316c1c18996b0ed5f642b83c90813571f0617c5e2517366dfb204df304dd77 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=9fb9da14-5985-423f-811a-a0c95887b9b8 name=kata-shim pid=62 source=shim
time="2019-03-15T15:31:59.647000126+08:00" level=info msg="copy stdout failed" container=7b051c5658e9f1738bfabec4a702e179595c667a91ee5e06ba3945f1ef55a539 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=ad62cac5-d1f6-40a2-89ec-ecf60e5a123d name=kata-shim pid=30 source=shim
time="2019-03-15T15:52:37.846463147+08:00" level=info msg="copy stdout failed" container=51e76f11d983ca6b2864b33bfe683e77917e40cf54ad2249b985511a1fef8027 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e0a78b61-1764-4787-84c1-7aa60e76e44e name=kata-shim pid=21 source=shim
time="2019-03-15T16:48:01.484014257+08:00" level=info msg="copy stdout failed" container=13ea0fd40c4a41157fa0eed0d4505e28d9d9e0fc01a31266961976980a2a3318 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=1347429d-14c4-4d7f-b443-3e4bf594c388 name=kata-shim pid=12 source=shim

Throttler logs

No recent throttler problems found in system journal.

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.9.4
 Git commit:      07f3374/1.13.1
 Built:           Fri Dec  7 16:13:51 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.7.5
 Git commit:      07f475d
 Built:           Wed Mar 13 07:02:01 2019
 OS/Arch:         linux/amd64
 Experimental:    false

Output of "docker info":

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 42
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-8:2-1444585-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 12.65 GB
 Data Space Total: 107.4 GB
 Data Space Available: 57.76 GB
 Metadata Space Used: 15.95 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.132 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.mt20180601.62.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 15.42 GiB
Name: localhost.localdomain
ID: CM5C:F347:A7JF:LJHE:HWGU:WFDM:GP2H:DN4B:VA4C:IXQG:BS4C:BTKK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

Output of "systemctl show docker":

Type=notify
Restart=on-abnormal
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=0
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestamp=五 2019-03-15 18:35:59 CST
WatchdogTimestampMonotonic=5550426602
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=2943
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=五 2019-03-15 18:35:54 CST
ExecMainStartTimestampMonotonic=5545331292
ExecMainExitTimestampMonotonic=0
ExecMainPID=2943
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd-current ; argv[]=/usr/bin/dockerd-current --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES ; ignore_errors=no ; start_time=[五 2019-03-15 18:35:54 CST] ; stop_time=[n/a] ; pid=2943 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=18446744073709551615
TasksCurrent=74
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=GOTRACEBACK=crash DOCKER_HTTP_HOST_COMPAT=1 PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
EnvironmentFile=/run/containers/registries.conf (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-storage (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=1048576
LimitAS=18446744073709551615
LimitNPROC=1048576
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=63048
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=basic.target docker-cleanup.timer
Wants=system.slice docker-storage-setup.service
RequiredBy=docker-cleanup.service
Conflicts=shutdown.target
Before=shutdown.target
After=network.target basic.target docker-storage-setup.service systemd-journald.socket system.slice
Documentation=http://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
UnitFileState=disabled
UnitFilePreset=disabled
InactiveExitTimestamp=五 2019-03-15 18:35:54 CST
InactiveExitTimestampMonotonic=5545331308
ActiveEnterTimestamp=五 2019-03-15 18:35:59 CST
ActiveEnterTimestampMonotonic=5550426624
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=五 2019-03-15 18:35:54 CST
ConditionTimestampMonotonic=5545307363
AssertTimestamp=五 2019-03-15 18:35:54 CST
AssertTimestampMonotonic=5545307363
Transient=no

No kubectl

Packages

qemu-img-1.5.3-156.el7_5.2.x86_64
kata-ksm-throttler-1.6.0~rc1.git+ce5b765-20.1.x86_64
qemu-kvm-1.5.3-156.el7_5.2.x86_64
qemu-vanilla-bin-2.11.2+git.0982a56a55-21.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7.noarch
qemu-lite-data-2.11.0+git.87517afd72-21.1.x86_64
qemu-kvm-common-1.5.3-156.el7_5.2.x86_64
qemu-lite-2.11.0+git.87517afd72-21.1.x86_64
kata-proxy-bin-1.6.0~rc1+git.2085829-18.1.x86_64
qemu-vanilla-data-2.11.2+git.0982a56a55-21.1.x86_64
kata-linux-container-4.19.24.25-20.1.x86_64
qemu-vanilla-2.11.2+git.0982a56a55-21.1.x86_64
kata-runtime-1.6.0~rc1+git.9f8d4e1-26.1.x86_64
qemu-lite-bin-2.11.0+git.87517afd72-21.1.x86_64
kata-shim-1.6.0~rc1+git.0f41347-16.1.x86_64
qemu-guest-agent-2.8.0-2.el7.x86_64
kata-proxy-1.6.0~rc1+git.2085829-18.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.5.x86_64
kata-shim-bin-1.6.0~rc1+git.0f41347-16.1.x86_64
kata-containers-image-1.6.0~rc1-17.1.x86_64

The text was updated successfully, but these errors were encountered:

jodh-intel · 2019-03-21T08:38:18Z

Hi @free2k - thanks for raising. A few questions:

What is the crawler image?
Does this work with runc?

grahamwhaley · 2019-03-21T10:05:48Z

Hi @free2k - you seem to be trying to run systemctl inside an un-priv container - containers don't run with an init system, so I'd not expect this to normally work :-)
https://forums.docker.com/t/systemctl-status-is-not-working-in-my-docker-container/9075
Is the -v /sys/fs/cgroup:/sys/fs/cgroup:ro on your command line maybe a docker hack to try and allow systemctl to access the host system?

What are you actually trying to achieve here? It feels like maybe you are trying to do some sort of docker privileged container access to the host.

/cc @amshinde for any thoughts around that mount etc.

free2k · 2019-03-21T11:11:13Z

Hi @jodh-intel

Crawler: v2 is based on the image built on centos7,this is my dockerfile

FROM centos:7

ENV container docker
MAINTAINER The CentOS Project cloud-ops@centos.org
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in ; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done);
echo "nameserver 114.114.114.114" > /etc/resolv.conf;
yum update;
cat /etc/resolv.conf;
rm -f /lib/systemd/system/multi-user.target.wants/;
rm -f /etc/systemd/system/.wants/;
rm -f /lib/systemd/system/local-fs.target.wants/;
rm -f /lib/systemd/system/sockets.target.wants/udev;
rm -f /lib/systemd/system/sockets.target.wants/initctl;
rm -f /lib/systemd/system/basic.target.wants/;
rm -f /lib/systemd/system/anaconda.target.wants/*;
yum install -y rp-pppoe vim net-tools less wget crontabs traceroute openssh-server openssh-clients javapackages-tools java-1.8.0-openjdk-devel.x86_64;
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config;
systemctl enable sshd;
rpm -qa | grep java | grep -v javapackages | xargs rpm -e --nodeps;
yum clean all;
rm -f /usr/lib/systemd/system/sysinit.target.wants/systemd-udev-trigger.service;
rm -f /usr/lib/systemd/system/sysinit.target.wants/systemd-udevd.service;
VOLUME ["/sys/fs/cgroup"]
CMD ["/usr/sbin/init"]

Runc can run systemd successfully，The container started by the following command can run sysmted inside the container.But adding the --runtime kata-runtime parameter will not run systemd

docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /var/run:/var/run --name kata-test --cpus 2 --memory 2G -itd crawler:v2

free2k · 2019-03-21T11:30:12Z

hi @grahamwhaley
Thank you for your answer, I want systemd to start as the first process.Does kata support the --privileged=true parameter? I am very confused about this.

grahamwhaley · 2019-03-21T11:39:49Z

Ah, right @free2k - I see, you want systemd inside the container. A question:

Q. Do you need the container to access the host system? That is, do you use the container to modify/control the host system?

I think the answer might be 'no', but when you are using runc, you need to allow the container access to the host cgroup volume so you can run systemd?

For Kata, we might have to do something different here - as Kata is running its own kernel inside a VM, what you probably would want is the container to have access to the cgroup volume of the VM kernel, and not the actual host system itself. I don't think we have a way to allow that today with Kata though.

For 'does kata support --privileged' - in our Limitations document we say 'no', but, I think that is not the whole story. afaik, kata does not support some host resource sharing. @amshinde , were you in the process of clarifying that in the Limitations document?

I'm not sure if we can support running systemd in the container right now, but would like to hear from @amshinde and @devimc at least. I think this is an interesting question :-) /cc @sboeuf @gnawux

free2k · 2019-03-21T11:53:53Z

In fact, I can run systemd in kata through some methods. Although this method looks very bad. Because systemd can't start is related to dbus, so I share the host's /run directory directly to the kata virtual machine. This way I can start systemd in the kata virtual machine.
I don't understand enough about dbus, but I think if the kata virtual machine has an independent dbus will solve this problem, instead of relying on the dbus of the host. @grahamwhaley

amshinde · 2019-03-22T00:06:01Z

@free2k I tried reproducing your issue with the Dockerfile contents you provided, but was not able to build it due to some errors in there. So I used the Dockerfile under Dockerfile for systemd base image at https://hub.docker.com/_/centos.
After building the image I was able not able to run systemctl even with a runc container!
@jodh-intel @grahamwhaley Can you try running the container with runc to confirm, I want to know if I am missing something.
I have also posted the image to dockerhub under my account: https://hub.docker.com/r/amshinde/centos-systemd.

Following that, I did follow the example posted on the centos docker page regarding systemd-enabled app container as:

FROM local/c7-systemd
RUN yum -y install httpd; yum clean all; systemctl enable httpd.service
EXPOSE 80
CMD ["/usr/sbin/init"]

Running that with sudo docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /tmp/$(mktemp -d):/run -p 80:80 local/c7-systemd-httpd worked for me with runc and kata as well. I was able to access the web page on port 80 of localhost with kata.

amshinde · 2019-03-22T00:11:12Z

For 'does kata support --privileged' - in our Limitations document we say 'no', but, I think that is not the whole story. afaik, kata does not support some host resource sharing. @amshinde , were you in the process of clarifying that in the Limitations document?

@grahamwhaley Done :) I have raised a PR for this. @free2k Take a look at this : kata-containers/documentation#408

@grahamwhaley For '/sys/fs/cgroup', we pass the guest side mount in case of system volumes, as it does not make sense to pass the host side volume. I am going to document this as well :)

free2k · 2019-03-22T10:26:44Z

@amshinde Thank you for your answer。
I have uploaded the image to the docker hub.
https://cloud.docker.com/repository/registry-1.docker.io/free2k/repository
You can use the following command to reproduce this problem.

docker run --runtime kata-runtime -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name kata-test --cpus 2 --memory 2G -itd free2k/repository:v1

But if you cancel the --runtime kata-runtime parameter, you can successfully run systemd

amshinde · 2019-03-22T17:55:10Z

@free2k Its interesting I am getting the same error with both kata and runc.
I was able to resolve it with adding flags --tmpfs /run --tmpfs /run/lock to the run command.
My environment is different through, I am seeing that you are running an old version of docker.
I would recommend you update your docker version if possible:
https://github.com/kata-containers/runtime/blob/master/versions.yaml#L208
For older versions of docker, you may need additional flag --security-opt seccomp=unconfined to your docker run command.

mcastelino · 2019-04-12T18:32:22Z

@free2k can you your setup with footloose containers. That may help

docker run --runtime=kata -d --mount type=tmpfs,destination=/run --mount type=tmpfs,destination=/run/lock --mount type=tmpfs,destination=/tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro quay.io/footloose/fedora29:latest /sbin/init

Ace-Tang · 2019-04-16T11:31:27Z

@free2k , could you please check mount -l | grep cgroup | grep systemd to see if the mount is rw, since systemd need write cgroup

amshinde mentioned this issue Apr 12, 2019

RFC: Kata + Footloose: VM Like containers in Kata #1531

Closed

jodh-intel added this to To do in Issue backlog Aug 10, 2020

jodh-intel closed this as completed Apr 7, 2021

Issue backlog automation moved this from To do to Done Apr 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to get D-Bus connection: Operation not permitted #1399

Failed to get D-Bus connection: Operation not permitted #1399

free2k commented Mar 21, 2019

jodh-intel commented Mar 21, 2019

grahamwhaley commented Mar 21, 2019

free2k commented Mar 21, 2019 •

edited

free2k commented Mar 21, 2019 •

edited

grahamwhaley commented Mar 21, 2019

free2k commented Mar 21, 2019 •

edited

amshinde commented Mar 22, 2019

amshinde commented Mar 22, 2019

free2k commented Mar 22, 2019

amshinde commented Mar 22, 2019

mcastelino commented Apr 12, 2019

Ace-Tang commented Apr 16, 2019

Failed to get D-Bus connection: Operation not permitted #1399

Failed to get D-Bus connection: Operation not permitted #1399

Comments

free2k commented Mar 21, 2019

Description of problem

Expected result

Actual result

Meta details

kata-env

Runtime config files

Runtime default config files

Runtime config file contents

KSM throttler

version

systemd service

Image details

Initrd details

Logfiles

Runtime logs

Proxy logs

Shim logs

Throttler logs

Container manager details

Docker

Packages

jodh-intel commented Mar 21, 2019

grahamwhaley commented Mar 21, 2019

free2k commented Mar 21, 2019 • edited

free2k commented Mar 21, 2019 • edited

grahamwhaley commented Mar 21, 2019

free2k commented Mar 21, 2019 • edited

amshinde commented Mar 22, 2019

amshinde commented Mar 22, 2019

free2k commented Mar 22, 2019

amshinde commented Mar 22, 2019

mcastelino commented Apr 12, 2019

Ace-Tang commented Apr 16, 2019

`kata-env`

free2k commented Mar 21, 2019 •

edited

free2k commented Mar 21, 2019 •

edited

free2k commented Mar 21, 2019 •

edited