Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

rpc error: code = Internal desc = Could not set name eth2 for interface file exists: unknown #785

Closed
fredbcode opened this issue Sep 25, 2018 · 38 comments

Comments

@fredbcode
Copy link

Description of problem

docker-compose up stop with an error message (works without kataruntime)

Expected result

All dockers image runs

Actual result

ERROR: for app_apache-rp_1 Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown

ERROR: for apache-rp Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.


Meta details

Running kata-collect-data.sh version 1.3.0-rc1 (commit 22aedc4) at 2018-09-25.12:56:13.961551045+0000.


Runtime is /usr/bin/kata-runtime.

kata-env

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.15"

[Runtime]
  Debug = true
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.0-rc1"
    Commit = "22aedc4"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  Msize9p = 8192
  Debug = true
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.0-rc1_agent_1ee972176ae.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.11-137.container"
  Parameters = ""

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-rc1-981fef4"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-rc1-9b2891c"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.15.0-34-generic"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Ubuntu"
    Version = "18.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz"

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found
Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's 
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons. 
# This flag prevents the block device from being passed to the hypervisor, 
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or 
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
# 
# Default false
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
# 
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes 
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default. 
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on 
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2018-09-13T22:26:12.472554196+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "25000"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.3.0-rc1-1ee972176ae437bcace0a37227818c506bb64ba9"
  agent-is-init-daemon: "no"

Initrd details

No initrd


Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-09-25T12:52:13.081180913Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18282 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:13.081273761Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="open /run/vc/sbs/3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b/devices.json: no such file or directory" name=kata-runtime pid=18282 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b sandboxid=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:13.112850265Z" level=debug arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18282 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:15.786608462Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18416 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=runtime
time="2018-09-25T12:52:15.851309108Z" level=error msg="Container 3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18451 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=runtime
time="2018-09-25T12:52:23.93719488Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc name=kata-runtime pid=18776 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:23.937275076Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc error="open /run/vc/sbs/ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc/devices.json: no such file or directory" name=kata-runtime pid=18776 sandbox=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc sandboxid=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:23.940444629Z" level=debug arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18776 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:25.041218059Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 name=kata-runtime pid=18850 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:25.041306973Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 error="open /run/vc/sbs/6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737/devices.json: no such file or directory" name=kata-runtime pid=18850 sandbox=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 sandboxid=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:25.044666392Z" level=debug arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18850 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:26.146000833Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b name=kata-runtime pid=18937 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:26.146090795Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b error="open /run/vc/sbs/ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b/devices.json: no such file or directory" name=kata-runtime pid=18937 sandbox=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b sandboxid=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:26.149340329Z" level=debug arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18937 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:27.234578298Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 name=kata-runtime pid=18936 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:27.234668383Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 error="open /run/vc/sbs/025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1/devices.json: no such file or directory" name=kata-runtime pid=18936 sandbox=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 sandboxid=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:27.237834807Z" level=debug arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18936 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:28.2194947Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb name=kata-runtime pid=19005 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:28.219589461Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb error="open /run/vc/sbs/7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb/devices.json: no such file or directory" name=kata-runtime pid=19005 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb sandboxid=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:28.222680814Z" level=debug arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19005 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:29.301948351Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=19580 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:29.302020019Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=19580 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:29.306542004Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19580 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:30.45857333Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd name=kata-runtime pid=19208 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:30.45867002Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd error="open /run/vc/sbs/1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd/devices.json: no such file or directory" name=kata-runtime pid=19208 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd sandboxid=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:30.465720646Z" level=debug arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19208 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:32.049678014Z" level=error msg="update interface request failed" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd error="rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4202940e0 0 0xc420017760 ether <nil> down 0 0 0 []}}: file exists" interface-requested="device:\"eth2\" name:\"eth2\" IPAddresses:<address:\"172.20.0.3\" mask:\"16\" > mtu:1500 hwAddr:\"02:42:ac:14:00:03\" " name=kata-runtime pid=19208 resulting-interface="<nil>" source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:32.284653706Z" level=error msg="rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4202940e0 0 0xc420017760 ether <nil> down 0 0 0 []}}: file exists" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd name=kata-runtime pid=19208 source=runtime
time="2018-09-25T12:53:35.291267859Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20048 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:53:35.357743544Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20083 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:53:36.805211646Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20162 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:53:36.805307288Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20162 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:53:36.809959751Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20162 source=virtcontainers subsystem=qemu
time="2018-09-25T12:54:42.635201905Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20323 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:54:42.701442185Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20358 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:54:44.157224658Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20435 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:54:44.157319126Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20435 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:54:44.161909512Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20435 source=virtcontainers subsystem=qemu
time="2018-09-25T12:55:49.999109408Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20581 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:55:50.065967266Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20618 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:55:51.501234794Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20695 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:55:51.501331066Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20695 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:55:51.505721051Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20695 source=virtcontainers subsystem=qemu
time="2018-09-25T12:55:52.881076812Z" level=error msg="update interface request failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201d45d4 0 0xc420017220 ether <nil> down 0 0 0 []}}: file exists" interface-requested="device:\"eth1\" name:\"eth1\" IPAddresses:<address:\"172.21.0.3\" mask:\"16\" > mtu:1500 hwAddr:\"02:42:ac:15:00:03\" " name=kata-runtime pid=20695 resulting-interface="<nil>" source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:55:53.024631795Z" level=error msg="rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201d45d4 0 0xc420017220 ether <nil> down 0 0 0 []}}: file exists" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20695 source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2018-09-25T12:52:29.346550913Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:29.347968943Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:29.721355457Z" level=info msg="time=\"2018-09-25T12:52:29.69625668Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb signal=\"child exited\" source=agent\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:30.41325028Z" level=info msg="[    0.555245] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.413968208Z" level=info msg="[    0.555811] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.46014495Z" level=info msg="[    0.602045] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.532208493Z" level=info msg="[    0.674066] systemd-journald[78]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.543204241Z" level=info msg="[    0.685121] systemd[79]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.565093226Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.568146089Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.945781594Z" level=info msg="time=\"2018-09-25T12:52:30.917286105Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=111 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:31.656728874Z" level=info msg="[    0.597650] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.657552474Z" level=info msg="[    0.598450] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.703661711Z" level=info msg="[    0.644517] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.749604961Z" level=info msg="[    0.690513] systemd[75]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.76842037Z" level=info msg="[    0.709265] systemd-journald[76]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.79259753Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.793060201Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:53:35.123132286Z" level=info msg="time=\"2018-09-25T12:53:35.093430369Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=111 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:35.125151466Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:53:35.39996404Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:53:37.791036025Z" level=info msg="[    0.430956] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.791736675Z" level=info msg="[    0.431591] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.836782316Z" level=info msg="[    0.476577] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.878559988Z" level=info msg="[    0.518363] systemd-journald[70]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.882172747Z" level=info msg="[    0.522104] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.928259714Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.930315784Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:38.329717686Z" level=info msg="time=\"2018-09-25T12:53:38.299431836Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:42.494115853Z" level=info msg="time=\"2018-09-25T12:54:42.462991419Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:42.495302946Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:54:42.738042246Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:54:45.149484374Z" level=info msg="[    0.435340] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.150155549Z" level=info msg="[    0.435947] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.193234336Z" level=info msg="[    0.478985] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.239850619Z" level=info msg="[    0.525611] systemd-journald[71]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.2449347Z" level=info msg="[    0.530747] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.280246221Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.283244161Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.690288717Z" level=info msg="time=\"2018-09-25T12:54:45.657952596Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:49.851166455Z" level=info msg="time=\"2018-09-25T12:55:49.817549873Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:49.8526652Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:55:50.106968285Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:55:52.49255434Z" level=info msg="[    0.434797] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.493227222Z" level=info msg="[    0.435424] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.535750235Z" level=info msg="[    0.477910] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.579513501Z" level=info msg="[    0.521724] systemd[71]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.611329036Z" level=info msg="[    0.553487] systemd-journald[80]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.621840806Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.62393186Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent

Shim logs

Recent shim problems found in system journal:

time="2018-09-25T12:52:15.667560942Z" level=info msg="copy stderr failed" container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="rpc error: code = Unknown desc = EOF" exec-id=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-shim pid=1 source=shim
time="2018-09-25T12:52:15.667588664Z" level=info msg="copy stdout failed" container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="rpc error: code = Unknown desc = EOF" exec-id=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-shim pid=1 source=shim
time="2018-09-25T12:53:35.120234685Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:53:35.120241713Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:54:42.492389169Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:54:42.492392873Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:55:49.848505925Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:55:49.848538129Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of "docker info":

Containers: 7
 Running: 5
 Paused: 0
 Stopped: 2
Images: 13
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: syslog
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: kata-runtime
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 22aedc4 (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-34-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.82GiB
Name: RNS-SV-KATA-001
ID: UWGY:MJA5:6VGJ:YAQY:X6F3:QRQL:JEE3:HWYM:23HC:MJHV:XSJR:J4UQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Tue 2018-09-25 09:49:43 UTC
WatchdogTimestampMonotonic=2971349640
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=6456
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=[not set]
GID=[not set]
NRestarts=0
ExecMainStartTimestamp=Tue 2018-09-25 09:49:42 UTC
ExecMainStartTimestampMonotonic=2970822338
ExecMainExitTimestampMonotonic=0
ExecMainPID=6456
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=[not set]
CPUUsageNSec=[not set]
TasksCurrent=245
IPIngressBytes=18446744073709551615
IPIngressPackets=18446744073709551615
IPEgressBytes=18446744073709551615
IPEgressPackets=18446744073709551615
Delegate=yes
DelegateControllers=cpu cpuacct io blkio memory devices pids
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=no
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=infinity
IPAccounting=no
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=infinity
LimitNPROCSoft=infinity
LimitMEMLOCK=16777216
LimitMEMLOCKSoft=16777216
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=257088
LimitSIGPENDINGSoft=257088
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend
AmbientCapabilities=
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=docker.socket system.slice sysinit.target
Wants=network-online.target
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=network-online.target system.slice basic.target firewalld.service systemd-journald.socket sysinit.target docker.socket
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/docker.service
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Tue 2018-09-25 09:49:43 UTC
StateChangeTimestampMonotonic=2971349643
InactiveExitTimestamp=Tue 2018-09-25 09:49:42 UTC
InactiveExitTimestampMonotonic=2970822385
ActiveEnterTimestamp=Tue 2018-09-25 09:49:43 UTC
ActiveEnterTimestampMonotonic=2971349643
ActiveExitTimestamp=Tue 2018-09-25 09:48:34 UTC
ActiveExitTimestampMonotonic=2902640442
InactiveEnterTimestamp=Tue 2018-09-25 09:48:35 UTC
InactiveEnterTimestampMonotonic=2903649196
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Tue 2018-09-25 09:49:42 UTC
ConditionTimestampMonotonic=2970821279
AssertTimestamp=Tue 2018-09-25 09:49:42 UTC
AssertTimestampMonotonic=2970821280
Transient=no
Perpetual=no
StartLimitIntervalUSec=1min
StartLimitBurst=3
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=1c51818194e7435ba5a2b7b6aa7cb9e7
CollectMode=inactive

No kubectl


Packages

Have dpkg
Output of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-containers-image|linux-container|qemu-)"":

ii  kata-containers-image                 1.3.0~rc1-34                      amd64        Kata containers image
ii  kata-linux-container                  4.14.67.11-137                    amd64        linux kernel optimised for container-like workloads.
ii  kata-proxy                            1.3.0~rc1+git.981fef4-34          amd64        
ii  kata-runtime                          1.3.0~rc1+git.22aedc4-49          amd64        
ii  kata-shim                             1.3.0~rc1+git.9b2891c-35          amd64        
ii  qemu-lite                             2.11.0+git.f886228056-50          amd64        linux kernel optimised for container-like workloads.
ii  qemu-vanilla                          2.11.2+git.a39e0b3e82-44          amd64        linux kernel optimised for container-like workloads.

No rpm


Perhaps related with #702

proxy.log
runtime.log
shim.log

@fredbcode
Copy link
Author

fredbcode commented Sep 26, 2018

No not related with #702

@jodh-intel
Copy link
Contributor

Hi @fredbcode - thanks for reporting. However, it's difficult to see what is happening because the proxy log errors captured when you ran kata-collect-data.sh show that agent debug was not enabled and the attached logs also do not have agent debug enabled.

If you are running on a dev / test box and are happy to lose your journal log, it would be useful if you could clear the journal and re-run:

@caoruidong
Copy link
Member

What are interfaces in all containers? Seems there is already a "eth2" interface?

@fredbcode
Copy link
Author

fredbcode commented Sep 26, 2018 via email

@fredbcode
Copy link
Author

@jodh-intel I will provide logs tomorrow

@jodh-intel
Copy link
Contributor

@fredbcode - thanks ;)

@fredbcode
Copy link
Author

fredbcode commented Sep 27, 2018

Haveged installed (see #702)

cat /proc/sys/kernel/random/entropy_avail
4066

Full log enabled and purged

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:94:66:16:91:ff brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/26 brd x.x.x.x scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::d294:66ff:fe16:91ff/64 scope link 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:94:66:16:92:00 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/30 brd x.x.x.x scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::d294:66ff:fe16:9200/64 scope link 
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:01 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:02 brd ff:ff:ff:ff:ff:ff
6: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:04 brd ff:ff:ff:ff:ff:ff 

Sorry I can't use kata-log-parser on this sever.

kata.txt
proxy.log
runtime.log
shim.log


ERROR: for app_viewbox_1  Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15Creating app_apache_1 ... error

ERROR: for app_apache_1  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for viewbox  Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for apache  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown

@fredbcode
Copy link
Author

fredbcode commented Sep 27, 2018

version: '2.1'
services:
  apache:
    build:
      context: "apache-rp"
    networks:
      - office
      - talkmost
      - viewbox
      - dokuwiki
    ports:
      - "x.x.x.x:443:443"
  office:
    image: registry.test/images/squid:current
    networks:
      - office
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - talkmost
    restart: unless-stopped

  viewbox:
    image: registry.test/images/nginx
    networks:
      - viewbox
    restart: unless-stopped

networks:
  office:
  talkmost:
  viewbox:
  dokuwiki:


docker-compose up 
Starting app_apache_1 ... 
app_viewbox_1 is up-to-date
Starting app_apache_1
app_office_1 is up-to-date
Starting app_apache_1 ... error

ERROR: for app_apache_1  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for apache  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.

docker-compose up without kata

docker-compose up
Creating network "app_dokuwiki" with the default driver
Creating network "app_viewbox" with the default driver
Creating network "app_office" with the default driver
Creating network "app_talkmost" with the default driver
Creating app_apache_1 ... 
Creating app_viewbox_1 ... 
Creating app_office_1 ... 
Creating app_talkmost_1 ... 
Creating app_viewbox_1
Creating app_office_1
Creating app_talkmost_1
Creating app_apache_1 ... done
Attaching to app_viewbox_1, app_talkmost_1, app_office_1, app_apache_1
office_1    | WARNING: Cannot write log file: stdio:/dev/stdout
office_1    | stdio:/dev/stdout: No such file or directory
office_1    |          messages will be sent to 'stderr'.
office_1    | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1    | WARNING: Cannot write log file: stdio:/dev/stdout
office_1    | stdio:/dev/stdout: No such file or directory
office_1    |          messages will be sent to 'stderr'.
office_1    | 2018/09/27 08:28:30| WARNING: Closing open FD    2
office_1    | 2018/09/27 08:28:30| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
office_1    | 2018/09/27 08:28:30| Service Name: squid
office_1    | 2018/09/27 08:28:30| Process ID 9
office_1    | 2018/09/27 08:28:30| Process Roles: master worker
office_1    | 2018/09/27 08:28:30| With 1048576 file descriptors available
office_1    | 2018/09/27 08:28:30| Initializing IP Cache...
office_1    | 2018/09/27 08:28:30| DNS Socket created at [::], FD 7
office_1    | 2018/09/27 08:28:30| DNS Socket created at 0.0.0.0, FD 8
office_1    | 2018/09/27 08:28:30| Adding domain rie from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Adding nameserver 127.0.0.11 from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Adding ndots 1 from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Logfile: opening log stdio:/dev/stdout
office_1    | 2018/09/27 08:28:30| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
office_1    | 2018/09/27 08:28:30| Store logging disabled
office_1    | 2018/09/27 08:28:30| Swap maxSize 0 + 262144 KB, estimated 20164 objects
office_1    | 2018/09/27 08:28:30| Target number of buckets: 1008
office_1    | 2018/09/27 08:28:30| Using 8192 Store buckets
office_1    | 2018/09/27 08:28:30| Max Mem  size: 262144 KB
office_1    | 2018/09/27 08:28:30| Max Swap size: 0 KB
office_1    | 2018/09/27 08:28:30| Using Least Load store dir selection
office_1    | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1    | 2018/09/27 08:28:30| Finished loading MIME types and icons.
office_1    | 2018/09/27 08:28:30| HTCP Disabled.
office_1    | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 11 to [::1]: (99) Cannot assign requested address
office_1    | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address
office_1    | 2018/09/27 08:28:30| ERROR: Failed to create helper child read FD: UDP[::1]
office_1    | 2018/09/27 08:28:30| Squid plugin modules loaded: 0
office_1    | 2018/09/27 08:28:30| Adaptation support is off.
office_1    | 2018/09/27 08:28:30| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 10 flags=9
office_1    | 2018/09/27 08:28:31| storeLateRelease: released 0 objects
apache_1    | [Thu Sep 27 08:28:32.487388 2018] [core:error] [pid 10:tid 139667229403008] (EAI 2)Name or service not known: AH00547: Could not resolve host name block.scn.rie.gouv.fr -- ignoring!
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0

This message is useless:
office_1 | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address
It's about squid and ipv6, same problem with another image without network needed ...

@fredbcode
Copy link
Author

Without
ports:
- "x.x.x.x:443:443"
And squid switched by ubuntu
Same punishment ...

@grahamwhaley
Copy link
Contributor

@amshinde any thoughts from the network side on the eth1/eth2 info here?

@caoruidong
Copy link
Member

Maybe related to different nic naming "ethXX" vs "enoXX".

@fredbcode
Copy link
Author

fredbcode commented Sep 27, 2018

I guess I found something very interesting !

version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
      - test2
  office:
    image: registry.test/images/ubuntu
    networks:
      - test
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - test1
    restart: unless-stopped

  talkmost1:
    image: registry.test/images/ubuntu
    networks:
      - test2
    restart: unless-stopped

networks:
  test:
  test1:
  test2:
docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ... 
Creating app_fred_1 ... 
Creating app_talkmost_1 ... 
Creating app_office_1 ... 
Creating app_talkmost1_1
Creating app_talkmost_1
Creating app_office_1
Creating app_fred_1 ... error

ERROR: for app_fred_1  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:Creating app_talkmost1_1 ... done

ERROR: for fred  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e05d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.

But with only two internal networks:

docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ... 
Creating app_talkmost_1 ... 
Creating app_fred_1 ... 
Creating app_office_1 ... 
Creating app_talkmost1_1
Creating app_fred_1
Creating app_talkmost_1
Creating app_office_1 ... done
Attaching to app_talkmost1_1, app_fred_1, app_talkmost_1, app_office_1
talkmost1_1  | WARNING: no logs are available with the 'syslog' log driver
fred_1       | WARNING: no logs are available with the 'syslog' log driver
talkmost_1   | WARNING: no logs are available with the 'syslog' log driver
office_1     | WARNING: no logs are available with the 'syslog' log driver
app_fred_1 exited with code 0
app_office_1 exited with code 0
app_talkmost1_1 exited with code 0
app_talkmost_1 exited with code 0
version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
#      - test2 **-> HERE**
  office:
    image: registry.test/images/ubuntu
    networks:
      - test
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - test1
    restart: unless-stopped

  talkmost1:
    image: registry.test/images/ubuntu
    networks:
      - test2
    restart: unless-stopped

networks:
  test:
  test1:
  test2:

I'm exactly in the same situation that before with issue #702 with Kata only one or two containers works well (but here my entropy is 4066)

My Dockerfile contains only a FROM to registry - ubuntu image -

@fredbcode
Copy link
Author

Of course I mean, max two containers works well -> containers with network in this case

@fredbcode
Copy link
Author

Syslog:

Sep 27 09:58:03 TEST kernel: [ 2720.132900] veth1d5f6d8: renamed from eth0
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 27 09:58:03 TEST kernel: [ 2720.202228] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST networkd-dispatcher[935]: WARNING:Unknown index 380 seen, reloading interface list
Sep 27 09:58:03 TEST kernel: [ 2720.220081] device veth9fd8479 left promiscuous mode
Sep 27 09:58:03 TEST kernel: [ 2720.220086] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: could not get ethtool features for veth1d5f6d8
Sep 27 09:58:03 TEST systemd-udevd[8922]: Could not set offload features of veth1d5f6d8: No such device
Sep 27 09:58:03 TEST networkd-dispatcher[935]: ERROR:Unknown interface index 380 seen even after reload
Sep 27 09:58:04 TEST systemd-networkd[971]: br-bc4629d62f80: Lost carrier

@jodh-intel
Copy link
Contributor

/cc @sboeuf, @amshinde, @mcastelino.

@sboeuf
Copy link

sboeuf commented Sep 28, 2018

@amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.

@grahamwhaley
Copy link
Contributor

Good question @sboeuf - and that is not going to surprise me. I think we have not tested Kata with docker compose much/enough. If this is the answer, can we check we have this in our Limitations do, and if not, add it.

@jodh-intel
Copy link
Contributor

+1. But also I wonder if there is a way for us to detect this scenario and fail elegantly?

@fredbcode
Copy link
Author

fredbcode commented Sep 28, 2018

Tried without ipv6 (I have a lot of warnings about) without success, someone can try to reproduce with this simple docker-compose ?

  1. docker system prune -a and reboot
  2. docker-compose up

Comment and uncomment a network and sometimes image works, sometimes not
When I have the error message a process kata-proxy still running (even after docker-compose down)

root 11013 1 0 09:25 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/proxy.sock -mux-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/kata.sock -sandbox c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a -log debug -agent-logs-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/console.sock

A kill command is needed

I'm pretty sure that there is also a timeout/exhausted resources issue here, because right after the same docker-compose can works (if kata-proxy is well dead)

version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
      - test2
      - test3
  fred1:
    image: registry.test/images/ubuntu
    networks:
      - test1

  fred2:
    image: registry.test/images/ubuntu
    networks:
      - test2

  fred3:
    image: registry.test/images/ubuntu
    networks:
      - test3

networks:
  test:
  test1:
  test2:
  test3:

A complete sequence:

docker-compose up  
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ... 
Creating app_fred3_1 ... 
Creating app_fred2_1 ... 
Creating app_fred_1 ... 
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... error

ERROR: for app_fred_1  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for fred  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.
root@TEST:/opt/app# killall kata-proxy
root@TEST:/opt/app# docker-compose down
Removing app_fred2_1 ... done
Removing app_fred_1  ... done
Removing app_fred3_1 ... done
Removing app_fred1_1 ... done
Removing network app_test1
Removing network app_test
Removing network app_test3
Removing network app_test2
root@TEST:/opt/app# docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ... 
Creating app_fred_1 ... 
Creating app_fred3_1 ... 
Creating app_fred2_1 ... 
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... done
Attaching to app_fred2_1, app_fred1_1, app_fred3_1, app_fred_1
fred2_1  | WARNING: no logs are available with the 'syslog' log driver
fred1_1  | WARNING: no logs are available with the 'syslog' log driver
fred3_1  | WARNING: no logs are available with the 'syslog' log driver
fred_1   | WARNING: no logs are available with the 'syslog' log driver
app_fred2_1 exited with code 0
app_fred1_1 exited with code 0
app_fred3_1 exited with code 0
app_fred_1 exited with code 0

@fredbcode
Copy link
Author

Here some tests I made, I'm trying to reproduce what docker-compose does:

A bash script with:

docker pull test.registry/images/ubuntu:16.04
docker pull test.registry/images/ubuntu:18.04

docker network create test1
docker network create test2
docker network create test3
docker network create test4
docker network create test5
docker network create test6

docker run -itd --net test1 --name testu1 test.registry/images/ubuntu:16.04
docker run -itd --net test2 --name testu2 test.registry/images/ubuntu:16.04
docker run -itd --net test3 --name testu3 test.registry/images/ubuntu:18.04
docker run -itd --net test4 --name testu4 test.registry/images/ubuntu:18.04
docker run -itd --net test5 --name testu5 test.registry/images/ubuntu:18.04
docker run -itd --net test6 --name testu6 test.registry/images/ubuntu:18.04

docker network connect test2 testu1
docker network connect test3 testu1
docker network connect test4 testu1
docker network connect test5 testu1
docker network connect test6 testu1

works well !

./test.sh 
16.04: Pulling from images/ubuntu
50513f4bae65: Pull complete 
854cd4146cdf: Pull complete 
3aa6df3c928d: Pull complete 
3bcf156a92e8: Pull complete 
10de78edcdec: Pull complete 
1a7902332604: Pull complete 
91da1c491187: Pull complete 
f6fe36c00817: Pull complete 
37141a578bef: Pull complete 
Digest: sha256:44f8fe6d233cb01205aa2d924a43fc6ebf9fc13d6d95748e092abb504148976d
Status: Downloaded newer image for test.registry/images/ubuntu:16.04
18.04: Pulling from images/ubuntu
09888d3b717d: Pull complete 
197bf3249b88: Pull complete 
e67b16720f7d: Pull complete 
a8357788d59a: Pull complete 
4c7f1cbe991a: Pull complete 
bcafe3c5b913: Pull complete 
6aa96a742a7e: Pull complete 
cd83ffe921b9: Pull complete 
Digest: sha256:7a2f553a2068d0c8ae95876bf22ff2594fda5a94386f54d686717112fed5c92c
Status: Downloaded newer image for test.registry/images/ubuntu:18.04
d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f
7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e
7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d
adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107
ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec
4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d
a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad
ba5cbee4c0e15f8eff79c97ad40fef8b77599e46a26d230d02e89aa18a28a410
f8d30f156a0a617451c2be18c7778a890617189e41b7dd7eb55172fd85223126
03ac095529891c067dd779ba31aed7e26104d118670857a7c95fc87c1318f96a
d420602a07b2a3f833616127ef99590417fb08d873f666bdedc022e6a0e36af2
02f778dd57feb0f6f55548ab33652eb76cb0e0f1bd548660f88ad942a8c966d3

docker inspect testu1

[
    {
        "Id": "a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad",
        "Created": "2018-10-04T08:01:07.019399113Z",
        "Path": "/bin/bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5918,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-10-04T08:01:09.127358791Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:c32058bebc576223f293a7f02e55b9752bb1045edcd3e01612689ad08aa5acd9",
        "ResolvConfPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hostname",
        "HostsPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hosts",
        "LogPath": "",
        "Name": "/testu1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "syslog-address": "udp://rns-sv-exploit-001.admin:514",
                    "tag": "{{.Name}}"
                }
            },
            "NetworkMode": "test1",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "kata-runtime",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18-init/diff:/var/lib/docker/overlay2/f0e1ddd045b79425da24fc000f12684e4d90397f1f57b5edf18d4c0fe4f86d6f/diff:/var/lib/docker/overlay2/6acf8d1674a94f2a9dafb9c417d33e9358922092198b84e0bc533984f14ddfbe/diff:/var/lib/docker/overlay2/c13b51b30f06f737308371141252b2223b15682e99c6bfb2970f91dab536d32c/diff:/var/lib/docker/overlay2/c52e355aa61eb36d271d4b4bca0cb3f8ad628e5eb3502b65ccd43892b2b5ca2e/diff:/var/lib/docker/overlay2/a5118628f9f643a10db743d454f360784f0c8fdbaed00553532daa0d11108674/diff:/var/lib/docker/overlay2/a8a0ef248466ed04d9ff93914df4502057d4e96df4092aee1e496b3e4a68e7e7/diff:/var/lib/docker/overlay2/74de80d7090ca44f286d7b900a7cbb75876479b76ce4d56dbaeb6cea6d2985e9/diff:/var/lib/docker/overlay2/4a22b5026632d77f15b1f446730d18e405eadb3d131a0752d81a7deaa0f9316d/diff:/var/lib/docker/overlay2/82d18854e6a535ecf63cade51ff697c7ef06b4c71f48680c12c11413ea91d3b5/diff",
                "MergedDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/merged",
                "UpperDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/diff",
                "WorkDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "a9ab1e0a9486",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "ArgsEscaped": true,
            "Image": "test.registry/images/ubuntu:16.04",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "build_date.ubuntu": "2018-07-26T08:51:08+00:00",
                "commit.ubuntu": "5857e7b5a75204a411678d8434a1f7e59bd553f9",
                "version.ubuntu": "20170915"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "608b8228387d4be30026671ed833c56e215ba2da95303ce7be17a32c563d458d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/608b8228387d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "test1": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f",
                    "EndpointID": "cfbca2d7d522508d1c6e4d335b065323c1d464d2c9e047352fb9e9e5cd6fb140",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02",
                    "DriverOpts": null
                },
                "test2": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e",
                    "EndpointID": "0f2a4c40447135e2cd701c9759f741182d41c8222a0460f057552e35ae02efe8",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:13:00:03",
                    "DriverOpts": null
                },
                "test3": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d",
                    "EndpointID": "b21bfb0e252502c193584fa59ab0425ba673d7a5d77f3e1de6d870c537fa44f9",
                    "Gateway": "172.20.0.1",
                    "IPAddress": "172.20.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:14:00:03",
                    "DriverOpts": null
                },
                "test4": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107",
                    "EndpointID": "0f24452f8d6038391110553bcaaff74698071d3a94c50420d6e2ca89b297d36e",
                    "Gateway": "172.21.0.1",
                    "IPAddress": "172.21.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:15:00:03",
                    "DriverOpts": null
                },
                "test5": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec",
                    "EndpointID": "48d5870d26bfdec9628bc6eb90a7e5ee242103c1b7cc0de5673ca39b249d3c26",
                    "Gateway": "172.22.0.1",
                    "IPAddress": "172.22.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:16:00:03",
                    "DriverOpts": null
                },
                "test6": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d",
                    "EndpointID": "078c6f9a2a40d66d45ca160e456635640d25fd4ac0f90bf2326657e05d171acf",
                    "Gateway": "172.23.0.1",
                    "IPAddress": "172.23.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:17:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

And FI same issue with:
https://github.com/docker/compose/releases/tag/1.23.0-rc1

Let me know if I can help you in any way

@fredbcode
Copy link
Author

fredbcode commented Oct 4, 2018

Latest dockerd build : docker version 18.06.1-ce, build e68fc7a

I have this message in syslog only when the bug occurs:

Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993840460Z" level=error msg="1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d cleanup: failed to delete container from containerd: no such container"
Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993900278Z" level=error msg="Handler for POST /v1.24/containers/1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:1b:00:02 broadcast|multicast 4098 0 0 0xc4201dc5d4 0 0xc4200172c0 ether down 0 0 0 []}}: file exists: unknown"

Docker in debug mode

DEBU[2018-10-04T09:15:16.646247764Z] Released address PoolID:LocalDefault/192.168.80.0/20, Address:192.168.80.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.80.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[2018-10-04T09:15:16.646674155Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc) 
DEBU[2018-10-04T09:15:16.648281440Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:15:16.706595927Z] Programming external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8) 
DEBU[2018-10-04T09:15:16.707361953Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.96.3, <nil>, true) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.707450537Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(082f8e75b4df, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c7**3722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.707475549Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(fred, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.790172433Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test2 
DEBU[2018-10-04T09:15:16.790205061Z] ReleaseAddress(LocalDefault/192.168.96.0/20, 192.168.96.3) 
DEBU[2018-10-04T09:15:16.790237425Z] Released address PoolID:LocalDefault/192.168.96.0/20, Address:192.168.96.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.96.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[2018-10-04T09:15:16.790621371Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8) 
DEBU[2018-10-04T09:15:16.792236669Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:15:16.830938459Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.112.3, <nil>, true) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.830990597Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(082f8e75b4df, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.831021853Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(fred, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.902267594Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test3 
DEBU[2018-10-04T09:15:16.902290954Z] ReleaseAddress(LocalDefault/192.168.112.0/20, 192.168.112.3) 
DEBU[2018-10-04T09:15:16.902317283Z] Released address PoolID:LocalDefault/192.168.112.0/20, Address:192.168.112.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.112.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
ERRO[2018-10-04T09:15:16.981674628Z] 082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498 cleanup: failed to delete container from containerd: no such container 
ERRO[2018-10-04T09:15:16.981733245Z] Handler for POST /v1.24/containers/082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:c0:a8:70:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown 

Same without error (with only two networks)

DEBU[2018-10-04T09:36:31.709013698Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(app_fred3_1, 192.168.224.2, <nil>, true) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.709067150Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(9c23b9033922, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.709088788Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(fred3, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.788133198Z] Releasing addresses for endpoint app_fred3_1's interface on network app_test3 
DEBU[2018-10-04T09:36:31.788181561Z] ReleaseAddress(LocalDefault/192.168.224.0/20, 192.168.224.2) 
DEBU[2018-10-04T09:36:31.788223099Z] Released address PoolID:LocalDefault/192.168.224.0/20, Address:192.168.224.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.224.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:3 
DEBU[0227] event published                               ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[0228] event published                               ns=moby topic="/tasks/create" type=containerd.events.TaskCreate
DEBU[2018-10-04T09:36:32.144238019Z] event                                         module=libcontainerd namespace=moby topic=/tasks/create
DEBU[0228] event published                               ns=moby topic="/tasks/start" type=containerd.events.TaskStart
DEBU[2018-10-04T09:36:32.201663950Z] event                                         module=libcontainerd namespace=moby topic=/tasks/start
DEBU[2018-10-04T09:36:32.247504205Z] Calling POST /v1.24/containers/8a30eeae9bb25bf39e3f4619470c79476d83da04afd302361d16ef8f659f8352/wait 
DEBU[2018-10-04T09:36:32.248237596Z] Calling POST /v1.24/containers/385d977c0a6baf95999e6f4da95678a05b83bb227f1596f0de226d378854bd5f/wait 
DEBU[2018-10-04T09:36:32.251847915Z] Calling POST /v1.24/containers/9c23b90339228c3b6315405fa16fb4f12aae83dbcbb8ce2a0c6eaee058d92b17/wait 
DEBU[2018-10-04T09:36:32.254077612Z] Calling POST /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/wait 
DEBU[2018-10-04T09:36:32.254794565Z] Calling GET /v1.24/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dapp%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D 
ERRO[0005] failed to kill init's children                error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437" pid=11055
DEBU[0228] event published                               ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2018-10-04T09:36:32.417929268Z] event                                         module=libcontainerd namespace=moby topic=/tasks/exit
INFO[0228] shim reaped                                   id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437
DEBU[0228] event published                               ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
DEBU[2018-10-04T09:36:32.565223072Z] event                                         module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2018-10-04T09:36:32.565267988Z] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[2018-10-04T09:36:32.565750086Z] Revoking external connectivity on endpoint app_fred2_1 (2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede) 
DEBU[2018-10-04T09:36:32.567607558Z] Calling GET /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/json 
DEBU[2018-10-04T09:36:32.568601235Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:36:32.633011532Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(app_fred2_1, 192.168.240.2, <nil>, true) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.633080246Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(4ce4a9495b1f, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.633102798Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(fred2, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.716011196Z] Releasing addresses for endpoint app_fred2_1's interface on network app_test2 
DEBU[2018-10-04T09:36:32.716044199Z] ReleaseAddress(LocalDefault/192.168.240.0/20, 192.168.240.2) 
DEBU[2018-10-04T09:36:32.716082403Z] Released address PoolID:LocalDefault/192.168.240.0/20, Address:192.168.240.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.240.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[0228] event published                               ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2018-10-04T09:36:33.000437658Z] Client context cancelled, stop sending events 

It works, although I have this in debug message (but nothing in prompt after docker-compose up)

ERRO[0005] failed to kill init's children error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437"

@Gabasjob
Copy link

Gabasjob commented Oct 9, 2018

Hello there.
Any update on this bug ?

Is there any workaround to use kata-runtime with docker-compose ?

@alebourdoulous
Copy link

Hello,
We look forward to hearing from you about this problem and hope to have a solution soon.
thank you

@grahamwhaley
Copy link
Contributor

Ping @amshinde @mcastelino - any thoughts, input or requests for more tests/info here?
I think @sboeuf had a pertinent question back up at #785 (comment) :

@amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.

@fredbcode
Copy link
Author

fredbcode commented Oct 10, 2018 via email

@Gabasjob
Copy link

Hello there,
here is some steps to reproduce the bug without docker-compose :

docker network create net1
docker network create net2
docker network create net3
docker create --name bb --network net1 busybox
docker network connect net2 bb
docker network connect net3 bb
docker start bb

output result :

Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:14:00:02 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown
Error: failed to start containers: bb

@amshinde
Copy link
Member

@sboeuf @grahamwhaley We do have a current limitation with docker-compose that causes issues in dns name resolution due to the use of dockerd. But the above issue looks unrelated.
I'll take a look to see whats going on.

@amshinde
Copy link
Member

@fredbcode @Gabasjob This patch to the runtime should solve your issue for now:
amshinde@d28b5b4

Can you give that a shot. I am planning to open a PR with that fix.

amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
@fredbcode
Copy link
Author

Great, can you share with us a binary please ?

@jodh-intel
Copy link
Contributor

@fredbcode - we don't really have the facility to do that. But all you'd need to do is:

  • Build the runtime with @amshinde's patch:
    $ go get -u github.com/kata-containers/runtime
    $ curl -sL https://github.com/amshinde/kata- runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd.patch | git apply
    $ make && sudo -E PATH=$PATH make install
    
  • Re-configure docker to use /usr/local/bin/kata-runtime that you just created rather than /usr/bin/kata-runtime.

@fredbcode
Copy link
Author

I will soon, thanks
@Gabasjob little "fix" for compilation

go get -u github.com/kata-containers/runtime/virtcontainers
cd yourpath/runtime
curl -sL https://github.com/amshinde/kata-runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd.patch | git apply
make && sudo -E PATH=$PATH make install

@Gabasjob
Copy link

@jodh-intel
Works well !

@grahamwhaley
Copy link
Contributor

Excellent! - thanks @Gabasjob , and kudos @amshinde
Note @Gabasjob - you will still have the dns resolution issue that @amshinde referenced above - which is listed (under swam) at:
https://github.com/kata-containers/documentation/blob/master/Limitations.md#docker-swarm-support
which points to some details at:
#175

@fredbcode
Copy link
Author

I also confirm, great !

@jodh-intel
Copy link
Contributor

Great - sounds like #814 will dtrt then ;)

@Gabasjob
Copy link

@grahamwhaley dns issue still present I'm affraid :
docker-compose.yml :

version: '2'
services:
  bb1:
    image: busybox
    networks:
      -  net1
    command: "ping bb2"
  bb2:
    image: busybox
    networks:
      -  net1
    command: "sleep 30"

networks:
  net1:

output without kata-runtime :

$ docker-compose up
Creating network "bugkata_net1" with the default driver
Recreating bugkata_bb2_1 ...
Recreating bugkata_bb1_1 ...
Recreating bugkata_bb1_1
Recreating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1  | PING bb2 (172.21.0.3): 56 data bytes  
bb1_1  | 64 bytes from 172.21.0.3: seq=0 ttl=64 time=1004.177 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=1 ttl=64 time=4.111 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=2 ttl=64 time=0.079 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=3 ttl=64 time=0.065 ms

output with kata-runtime :

$ docker-compose up
Creating network "bugkata_net1" with the default driver
Creating bugkata_bb2_1 ...
Creating bugkata_bb1_1 ...
Creating bugkata_bb1_1
Creating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1  | ping: bad address 'bb2'
bugkata_bb1_1 exited with code 1

@grahamwhaley
Copy link
Contributor

@Gabasjob yep - that is a known limitation.
/cc @amshinde @mcastelino - do we have any known workarounds, or long term plans on that?

amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
(cherry picked from commit 8f1b28d)
amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
(cherry picked from commit 8f1b28d)
amshinde added a commit to amshinde/kata-runtime that referenced this issue Oct 11, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
(cherry picked from commit 8f1b28d)
@amshinde amshinde reopened this Oct 11, 2018
@amshinde
Copy link
Member

@grahamwhaley Unfortunately there are no workarounds for this due to the way name resolution is handled in docker for custom networks.
I'll take a look at this with @mcastelino to come up with a plan.

zklei pushed a commit to zklei/runtime that referenced this issue Nov 22, 2018
Sort endpoints by name to control the order in which
they are passed to the VM as the interface name inside
the VM depends on the order in which it is passed.

Long term we should come up with a more robust approach.

Fixes kata-containers#785

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
@jodh-intel jodh-intel added this to To do in Issue backlog Aug 10, 2020
egernst pushed a commit to egernst/runtime that referenced this issue Feb 9, 2021
Makefile: do not use LDFLAGS to avoid environment contamination
Issue backlog automation moved this from To do to Done Apr 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Issue backlog
  
Done
Development

No branches or pull requests

8 participants