Skip to content
This repository has been archived by the owner on Mar 28, 2018. It is now read-only.

Failed to run container in cor runtime using Hyper-V #666

Open
amsokol opened this issue Feb 12, 2017 · 12 comments
Open

Failed to run container in cor runtime using Hyper-V #666

amsokol opened this issue Feb 12, 2017 · 12 comments
Labels
CC 2.1 type:bug
Projects
2.1.x Stable

Comments

@amsokol
Copy link

amsokol commented Feb 12, 2017 •
edited
Loading

Dear All,

I get the error when I try to run container in Clear Linux (both 13260 and 13330 releases).
Clear Linux is deployed from clear-13330(13260)-hyperv.vhdx.gz with enabled nested virtualization in Hyper-V on Windows 10 x64 (with latest updates).

ERROR:

docker pull bash

Using default tag: latest
latest: Pulling from library/bash
0a8490d0dfd3: Pull complete
12503a53c30a: Pull complete
d3fce0ccf57d: Pull complete
Digest: sha256:d7983e735b8a07443c6c4f87c9dc3386dc00ddbcf2930d44ba24aa958c40967c
Status: Downloaded newer image for bash:latest

docker run -it bash

docker: Error response from daemon: containerd: container did not start before the specified timeout.

Here is my environment (for 13330 release):

docker info

Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 1.12.6
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay null host
Swarm: inactive
Runtimes: cor runc
Default Runtime: cor
Security Options:
Kernel Version: 4.9.7-80.hyperv
Operating System: Clear Linux Software for Intel Architecture
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.855 GiB
Name: docker-icc-host01.amsokol.me
ID: DCO5:TE66:CM56:YU3V:OCB6:E3TP:ADY2:QJWE:TUDJ:2GCD:QV3X:7BOB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8

./clear-linux-check-config.sh container

Checking if host is capable of running Clear Linux* OS for Intel® Architecture in a container
SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
SUCCESS: Virtualisation support (vmx)
SUCCESS: Kernel module kvm
SUCCESS: Kernel module kvm_intel
SUCCESS: Nested KVM support

When I use runc instead of cor everything is OK.
Any ideas?

Thanks,
Alex
@amsokol
Copy link
Author

amsokol commented Feb 12, 2017 •
edited
Loading

May be it helps

systemctl status docker-cor

docker-cor.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker-cor.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2017-02-12 08:56:11 UTC; 25min ago
Docs: https://docs.docker.com
Main PID: 369 (dockerd)
Tasks: 34
Memory: 281.9M
CPU: 1h 25min 47.074s
CGroup: /system.slice/docker-cor.service
├─369 /usr/bin/dockerd -H fd:// --storage-driver=overlay --add-runtime cor=cc-oci-runtime --default-runtime=cor
├─378 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
├─597 /usr/bin/qemu-lite-system-x86_64 -name 903cf59b790b -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm -device nvdimm,memdev=mem0,id=nv0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-13280-containers.img,s
ize=235929600 -m 2G,slots=2,maxmem=3G -kernel /usr/share/clear-containers/vmlinux-4.9.4-55.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.dir
ect=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=syste
md-networkd.socket systemd.show_status=false cryptomgr.notests ip=::::::439c63724d53::off:: -device virtio-9p-pci,fsdev=workload9p,mount_tag=rootfs -fsdev local,id=workload9p,path=/var/lib/docker/overlay/b7e59c0207e41ba52ed490a4cef91797513315be0759b2a62
ebb03918bb6950d/merged,security_model=none -smp 2,sockets=1,cores=2,threads=1 -cpu host -rtc base=utc,driftfix=slew -no-user-config -nodefaults -global kvm-pit.lost_tick_policy=discard -device virtio-serial-pci,id=virtio-serial0 -device virtconsole,char
dev=charconsole0,id=console0 -chardev socket,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/console.sock,server,nowait,id=charconsole0,signal=off -chardev socket,id=procsock,path=/var/run/cc-oci-runtime/439
c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/process.sock,server,nowait -chardev socket,id=charch0,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/ga-ctl.sock,server,nowait -device virtseria
lport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/ga-tty.sock,server,nowait -device virtserialport
,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -uuid d663732e-4bf1-4065-a4e3-903cf59b790b -qmp unix:/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/hypervisor.sock,server,nowait -n
ographic -vga none -netdev tap,ifname=ceth0,script=no,downscript=no,id=ceth0,vhost=on -device driver=virtio-net-pci,bus=/pci-lite-host/pcie.0,addr=8,netdev=ceth0,mac=02:42:ac:11:00:02
└─715 /usr/bin/qemu-lite-system-x86_64 -name 467e7b4d5a7d -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm -device nvdimm,memdev=mem0,id=nv0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-13280-containers.img,s
ize=235929600 -m 2G,slots=2,maxmem=3G -kernel /usr/share/clear-containers/vmlinux-4.9.4-55.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.dir
ect=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=syste
md-networkd.socket systemd.show_status=false cryptomgr.notests ip=::::::439c63724d53::off:: -device virtio-9p-pci,fsdev=workload9p,mount_tag=rootfs -fsdev local,id=workload9p,path=/var/lib/docker/overlay/b7e59c0207e41ba52ed490a4cef91797513315be0759b2a62
ebb03918bb6950d/merged,security_model=none -smp 2,sockets=1,cores=2,threads=1 -cpu host -rtc base=utc,driftfix=slew -no-user-config -nodefaults -global kvm-pit.lost_tick_policy=discard -device virtio-serial-pci,id=virtio-serial0 -device virtconsole,char
dev=charconsole0,id=console0 -chardev socket,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/console.sock,server,nowait,id=charconsole0,signal=off -chardev socket,id=procsock,path=/var/run/cc-oci-runtime/439
c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/process.sock,server,nowait -chardev socket,id=charch0,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/ga-ctl.sock,server,nowait -device virtseria
lport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/ga-tty.sock,server,nowait -device virtserialport
,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -uuid b3a7b931-3df5-49da-ac0f-467e7b4d5a7d -qmp unix:/var/run/cc-oci-runtime/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/hypervisor.sock,server,nowait -n
ographic -vga none -netdev tap,ifname=ceth0,script=no,downscript=no,id=ceth0,vhost=on -device driver=virtio-net-pci,bus=/pci-lite-host/pcie.0,addr=8,netdev=ceth0,mac=02:42:ac:11:00:02

Feb 12 08:56:11 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T08:56:11.891314300Z" level=info msg="API listen on /var/run/docker.sock"
Feb 12 08:56:11 docker-icc-host01.amsokol.me systemd[1]: Started Docker Application Container Engine.
Feb 12 08:59:35 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T08:59:35.5833013Z" level=error msg="containerd: start container" error="containerd: container did not start before the specified timeout" id=439c63724d53872d8dfc1afa4d7bbc10b71b
4aa6cc34666d303d8a5aa53fa007
Feb 12 08:59:35 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T08:59:35.583822700Z" level=error msg="Create container failed with error: containerd: container did not start before the specified timeout"
Feb 12 08:59:35 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T08:59:35.805368400Z" level=error msg="Handler for POST /v1.24/containers/439c63724d53872d8dfc1afa4d7bbc10b71b4aa6cc34666d303d8a5aa53fa007/start returned error: containerd: conta
iner did not start before the specified timeout"
Feb 12 09:03:12 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T09:03:12.5843461Z" level=error msg="containerd: start container" error="containerd: container did not start before the specified timeout" id=439c63724d53872d8dfc1afa4d7bbc10b71b
4aa6cc34666d303d8a5aa53fa007
Feb 12 09:03:12 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T09:03:12.586589400Z" level=error msg="Create container failed with error: containerd: container did not start before the specified timeout"
Feb 12 09:03:12 docker-icc-host01.amsokol.me dockerd[369]: time="2017-02-12T09:03:12.877296400Z" level=error msg="Handler for POST /v1.24/containers/439c63724d53/start returned error: containerd: container did not start before the specified timeout"

@amsokol amsokol changed the title Failed to start container in cor environment Failed to start container in cor runtime Feb 12, 2017
@amsokol amsokol changed the title Failed to start container in cor runtime Failed to run container in cor runtime Feb 12, 2017
@amsokol
Copy link
Author

amsokol commented Feb 14, 2017 •
edited
Loading

The same error is in release 13350

@devimc
Copy link
Contributor

devimc commented Feb 15, 2017

Hi @amsokol
Could you please modify your docker service file in order to point the runtime to /usr/bin/cc-oci-runtime.sh and uncomment next lines of that script?

runtime_args="$runtime_args --global-log=\"$global_log\""
runtime_args="$runtime_args --debug"

run your tests again and a log file will be create

/run/cc-oci-runtime/cc-oci-runtime.log

please attach it

@amsokol
Copy link
Author

amsokol commented Feb 16, 2017

Dear @devimc,

Done:

2017-02-16T15:05:45.773284Z:517:cc-oci-runtime:debug:called as: /usr/bin/cc-oci-runtime create --bundle /var/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674 --console /dev/pts/1 --pid-file /run/docker/libcontainerd/containerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/init/pid dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674
2017-02-16T15:05:45.773431Z:517:cc-oci-runtime:debug:path '/var/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674' resolved to '/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674'
2017-02-16T15:05:45.773463Z:517:cc-oci-runtime:debug:using config_file /run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/config.json
2017-02-16T15:05:45.773817Z:517:cc-oci-runtime:debug:path '/var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged' resolved to '/var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged'
2017-02-16T15:05:45.773881Z:517:cc-oci-runtime:debug:Reading VM configuration from /usr/share/defaults/cc-oci-runtime/vm.json
2017-02-16T15:05:45.774304Z:517:cc-oci-runtime:debug:path '/usr/bin/qemu-lite-system-x86_64' resolved to '/usr/bin/qemu-lite-system-x86_64'
2017-02-16T15:05:45.774698Z:517:cc-oci-runtime:debug:path '/usr/share/clear-containers/clear-containers.img' resolved to '/usr/share/clear-containers/clear-13280-containers.img'
2017-02-16T15:05:45.774739Z:517:cc-oci-runtime:debug:path '/usr/share/clear-containers/vmlinux.container' resolved to '/usr/share/clear-containers/vmlinux-4.9.4-55.container'
2017-02-16T15:05:45.774783Z:517:cc-oci-runtime:debug:OCI spec versions: config=1.0.0-rc2-dev, runtime=1.0.0-rc1
2017-02-16T15:05:45.774813Z:517:cc-oci-runtime:debug:creating directory /var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674
2017-02-16T15:05:45.774840Z:517:cc-oci-runtime:debug:setting up namespaces
2017-02-16T15:05:45.774933Z:517:cc-oci-runtime:debug:created mount namespace
2017-02-16T15:05:45.775320Z:517:cc-oci-runtime:debug:created network namespace
2017-02-16T15:05:45.775345Z:517:cc-oci-runtime:debug:ignoring uts namespace request
2017-02-16T15:05:45.775361Z:517:cc-oci-runtime:debug:ignoring pid namespace request
2017-02-16T15:05:45.775377Z:517:cc-oci-runtime:debug:ignoring ipc namespace request
2017-02-16T15:05:45.775392Z:517:cc-oci-runtime:debug:finished namespace setup
2017-02-16T15:05:45.775408Z:517:cc-oci-runtime:debug:ignoring mount /proc
2017-02-16T15:05:45.775424Z:517:cc-oci-runtime:debug:ignoring mount /dev
2017-02-16T15:05:45.775439Z:517:cc-oci-runtime:debug:ignoring mount /dev/pts
2017-02-16T15:05:45.775455Z:517:cc-oci-runtime:debug:ignoring mount /sys
2017-02-16T15:05:45.775471Z:517:cc-oci-runtime:debug:ignoring mount /sys/fs/cgroup
2017-02-16T15:05:45.775486Z:517:cc-oci-runtime:debug:ignoring mount /dev/mqueue
2017-02-16T15:05:45.775516Z:517:cc-oci-runtime:debug:mounting /var/lib/docker/containers/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/resolv.conf of type bind onto /var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged/etc/resolv.conf with options '' and flags 0x45000
2017-02-16T15:05:45.775571Z:517:cc-oci-runtime:debug:mounting /var/lib/docker/containers/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/hostname of type bind onto /var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged/etc/hostname with options '' and flags 0x45000
2017-02-16T15:05:45.775615Z:517:cc-oci-runtime:debug:mounting /var/lib/docker/containers/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/hosts of type bind onto /var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged/etc/hosts with options '' and flags 0x45000
2017-02-16T15:05:45.775647Z:517:cc-oci-runtime:debug:ignoring mount /dev/shm
2017-02-16T15:05:45.775669Z:517:cc-oci-runtime:debug:connecting to proxy cc-proxy
2017-02-16T15:05:45.778861Z:517:cc-oci-runtime:debug:connected to proxy socket /var/run/cc-oci-runtime/proxy.sock
2017-02-16T15:05:45.779001Z:517:cc-oci-runtime:debug:hypervisor child pid is 518
2017-02-16T15:05:45.779083Z:518:cc-oci-runtime:debug:disconnecting from proxy
2017-02-16T15:05:45.779158Z:517:cc-oci-runtime:debug:shim process running with pid 519
2017-02-16T15:05:45.779191Z:518:cc-oci-runtime:debug:reading hypervisor command-line length from pipe
2017-02-16T15:05:45.779245Z:519:cc-oci-runtime:debug:shim child waiting for proxy socket fd on fd 6
2017-02-16T15:05:45.779362Z:517:cc-oci-runtime:debug:created state file /var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/state.json
2017-02-16T15:05:45.852103Z:517:cc-oci-runtime:debug:building hypervisor command-line
2017-02-16T15:05:45.852321Z:517:cc-oci-runtime:debug:Discovering container interfaces
2017-02-16T15:05:45.852384Z:517:cc-oci-runtime:debug:Interface := [lo]
2017-02-16T15:05:45.852427Z:517:cc-oci-runtime:debug:Interface := [tunl0]
2017-02-16T15:05:45.852466Z:517:cc-oci-runtime:debug:Interface := [eth0]
2017-02-16T15:05:45.852505Z:517:cc-oci-runtime:debug:Interface := [lo]
2017-02-16T15:05:45.852543Z:517:cc-oci-runtime:debug:Interface := [eth0]
2017-02-16T15:05:45.852609Z:517:cc-oci-runtime:debug:IP := [172.17.0.2]
2017-02-16T15:05:45.852653Z:517:cc-oci-runtime:debug:IP := [255.255.0.0]
2017-02-16T15:05:45.852695Z:517:cc-oci-runtime:debug:netlink_get_default_gw
2017-02-16T15:05:45.852759Z:517:cc-oci-runtime:debug:table=254
2017-02-16T15:05:45.852807Z:517:cc-oci-runtime:debug:IP := [172.17.0.1]
2017-02-16T15:05:45.852848Z:517:cc-oci-runtime:debug:gateway : 172.17.0.1
2017-02-16T15:05:45.852948Z:517:cc-oci-runtime:debug:ifname=eth0
2017-02-16T15:05:45.852995Z:517:cc-oci-runtime:debug:table=254
2017-02-16T15:05:45.853036Z:517:cc-oci-runtime:debug:IP := [172.17.0.0]
2017-02-16T15:05:45.853074Z:517:cc-oci-runtime:debug:destination : 172.17.0.0
2017-02-16T15:05:45.853120Z:517:cc-oci-runtime:debug:ifname=eth0
2017-02-16T15:05:45.853161Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853199Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853238Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853275Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853313Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853351Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853439Z:517:cc-oci-runtime:debug:table=255
2017-02-16T15:05:45.853491Z:517:cc-oci-runtime:debug:[1] networks discovered
2017-02-16T15:05:45.858748Z:517:cc-oci-runtime:debug:netlink_link_add_bridge beth0
2017-02-16T15:05:45.859118Z:517:cc-oci-runtime:debug:netlink_link_set_addr eth0
2017-02-16T15:05:45.859188Z:517:cc-oci-runtime:debug:macaddr 02:00:ca:fe:00:00
2017-02-16T15:05:45.859289Z:517:cc-oci-runtime:debug:netlink_link_set_master 3 4
2017-02-16T15:05:45.859485Z:517:cc-oci-runtime:debug:netlink_link_set_master 5 4
2017-02-16T15:05:45.859678Z:517:cc-oci-runtime:debug:netlink_link_enable[1] ceth0
2017-02-16T15:05:45.859767Z:517:cc-oci-runtime:debug:netlink_link_enable[1] eth0
2017-02-16T15:05:45.859821Z:517:cc-oci-runtime:debug:netlink_link_enable[1] beth0
2017-02-16T15:05:45.859912Z:517:cc-oci-runtime:debug:network configuration complete
2017-02-16T15:05:45.859958Z:517:cc-oci-runtime:debug:PCI Offset used for network: 8
2017-02-16T15:05:45.860033Z:517:cc-oci-runtime:debug:using /usr/share/defaults/cc-oci-runtime/hypervisor.args
2017-02-16T15:05:45.860838Z:517:cc-oci-runtime:debug:guest agent ctl socket: /var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-ctl.sock
2017-02-16T15:05:45.860918Z:517:cc-oci-runtime:debug:guest agent tty socket: /var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-tty.sock
2017-02-16T15:05:45.861084Z:517:cc-oci-runtime:debug:checking child setup (blocking)
2017-02-16T15:05:45.861172Z:518:cc-oci-runtime:debug:reading hypervisor command-line from pipe
2017-02-16T15:05:45.861289Z:518:cc-oci-runtime:debug:running command:
2017-02-16T15:05:45.861350Z:518:cc-oci-runtime:debug:arg: '/usr/bin/qemu-lite-system-x86_64'
2017-02-16T15:05:45.861429Z:518:cc-oci-runtime:debug:arg: '-name'
2017-02-16T15:05:45.861479Z:518:cc-oci-runtime:debug:arg: '2918c082584d'
2017-02-16T15:05:45.861520Z:518:cc-oci-runtime:debug:arg: '-machine'
2017-02-16T15:05:45.861561Z:518:cc-oci-runtime:debug:arg: 'pc-lite,accel=kvm,kernel_irqchip,nvdimm'
2017-02-16T15:05:45.861601Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.861641Z:518:cc-oci-runtime:debug:arg: 'nvdimm,memdev=mem0,id=nv0'
2017-02-16T15:05:45.861688Z:518:cc-oci-runtime:debug:arg: '-object'
2017-02-16T15:05:45.861728Z:518:cc-oci-runtime:debug:arg: 'memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-13280-containers.img,size=235929600'
2017-02-16T15:05:45.861778Z:518:cc-oci-runtime:debug:arg: '-m'
2017-02-16T15:05:45.861819Z:518:cc-oci-runtime:debug:arg: '2G,slots=2,maxmem=3G'
2017-02-16T15:05:45.861869Z:518:cc-oci-runtime:debug:arg: '-kernel'
2017-02-16T15:05:45.861909Z:518:cc-oci-runtime:debug:arg: '/usr/share/clear-containers/vmlinux-4.9.4-55.container'
2017-02-16T15:05:45.861957Z:518:cc-oci-runtime:debug:arg: '-append'
2017-02-16T15:05:45.862004Z:518:cc-oci-runtime:debug:arg: 'root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests ip=::::::dfd9317bddbd::off::'
2017-02-16T15:05:45.862065Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.862106Z:518:cc-oci-runtime:debug:arg: 'virtio-9p-pci,fsdev=workload9p,mount_tag=rootfs'
2017-02-16T15:05:45.862147Z:518:cc-oci-runtime:debug:arg: '-fsdev'
2017-02-16T15:05:45.862187Z:518:cc-oci-runtime:debug:arg: 'local,id=workload9p,path=/var/lib/docker/overlay/bc46df8f27b7a173fbc5ac61d35da32ec100d37a70f28a8a12ac23fc5bb59317/merged,security_model=none'
2017-02-16T15:05:45.862239Z:518:cc-oci-runtime:debug:arg: '-smp'
2017-02-16T15:05:45.862279Z:518:cc-oci-runtime:debug:arg: '2,sockets=1,cores=2,threads=1'
2017-02-16T15:05:45.862319Z:518:cc-oci-runtime:debug:arg: '-cpu'
2017-02-16T15:05:45.862358Z:518:cc-oci-runtime:debug:arg: 'host'
2017-02-16T15:05:45.862397Z:518:cc-oci-runtime:debug:arg: '-rtc'
2017-02-16T15:05:45.862435Z:518:cc-oci-runtime:debug:arg: 'base=utc,driftfix=slew'
2017-02-16T15:05:45.862475Z:518:cc-oci-runtime:debug:arg: '-no-user-config'
2017-02-16T15:05:45.862513Z:518:cc-oci-runtime:debug:arg: '-nodefaults'
2017-02-16T15:05:45.862552Z:518:cc-oci-runtime:debug:arg: '-global'
2017-02-16T15:05:45.862591Z:518:cc-oci-runtime:debug:arg: 'kvm-pit.lost_tick_policy=discard'
2017-02-16T15:05:45.862630Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.862670Z:518:cc-oci-runtime:debug:arg: 'virtio-serial-pci,id=virtio-serial0'
2017-02-16T15:05:45.862709Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.862747Z:518:cc-oci-runtime:debug:arg: 'virtconsole,chardev=charconsole0,id=console0'
2017-02-16T15:05:45.862788Z:518:cc-oci-runtime:debug:arg: '-chardev'
2017-02-16T15:05:45.862827Z:518:cc-oci-runtime:debug:arg: 'socket,path=/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/console.sock,server,nowait,id=charconsole0,signal=off'
2017-02-16T15:05:45.862870Z:518:cc-oci-runtime:debug:arg: '-chardev'
2017-02-16T15:05:45.862911Z:518:cc-oci-runtime:debug:arg: 'socket,id=procsock,path=/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/process.sock,server,nowait'
2017-02-16T15:05:45.862953Z:518:cc-oci-runtime:debug:arg: '-chardev'
2017-02-16T15:05:45.862993Z:518:cc-oci-runtime:debug:arg: 'socket,id=charch0,path=/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-ctl.sock,server,nowait'
2017-02-16T15:05:45.863040Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.863080Z:518:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0'
2017-02-16T15:05:45.863123Z:518:cc-oci-runtime:debug:arg: '-chardev'
2017-02-16T15:05:45.863163Z:518:cc-oci-runtime:debug:arg: 'socket,id=charch1,path=/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-tty.sock,server,nowait'
2017-02-16T15:05:45.863204Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.863242Z:518:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1'
2017-02-16T15:05:45.863282Z:518:cc-oci-runtime:debug:arg: '-uuid'
2017-02-16T15:05:45.863321Z:518:cc-oci-runtime:debug:arg: 'c18813e0-4c10-4ed7-a2e0-2918c082584d'
2017-02-16T15:05:45.863360Z:518:cc-oci-runtime:debug:arg: '-qmp'
2017-02-16T15:05:45.863399Z:518:cc-oci-runtime:debug:arg: 'unix:/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/hypervisor.sock,server,nowait'
2017-02-16T15:05:45.863441Z:518:cc-oci-runtime:debug:arg: '-nographic'
2017-02-16T15:05:45.863482Z:518:cc-oci-runtime:debug:arg: '-vga'
2017-02-16T15:05:45.863520Z:518:cc-oci-runtime:debug:arg: 'none'
2017-02-16T15:05:45.863558Z:518:cc-oci-runtime:debug:arg: '-netdev'
2017-02-16T15:05:45.863596Z:518:cc-oci-runtime:debug:arg: 'tap,ifname=ceth0,script=no,downscript=no,id=ceth0,vhost=on'
2017-02-16T15:05:45.863639Z:518:cc-oci-runtime:debug:arg: '-device'
2017-02-16T15:05:45.863678Z:518:cc-oci-runtime:debug:arg: 'driver=virtio-net-pci,bus=/pci-lite-host/pcie.0,addr=8,netdev=ceth0,mac=02:42:ac:11:00:02'
2017-02-16T15:05:45.872117Z:517:cc-oci-runtime:debug:child setup successful
2017-02-16T15:05:45.942172Z:517:cc-oci-runtime:debug:communicating with proxy
2017-02-16T15:05:45.942267Z:517:cc-oci-runtime:debug:sending message (length 447) to proxy socket
2017-02-16T15:05:45.942320Z:517:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hello","data":{"containerId":"dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674","ctlSerial":"/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-ctl.sock","ioSerial":"/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/ga-tty.sock","console":"/var/run/cc-oci-runtime/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/console.sock"}}
2017-02-16T15:05:46.392015Z:517:cc-oci-runtime:debug:proxy msg length: 37
2017-02-16T15:05:46.392082Z:517:cc-oci-runtime:debug:message read from proxy socket: {"success":true,"data":{"version":1}}
2017-02-16T15:05:46.392175Z:517:cc-oci-runtime:debug:msg received: {"success":true,"data":{"version":1}}
2017-02-16T15:05:46.392386Z:517:cc-oci-runtime:debug:communicating with proxy
2017-02-16T15:05:46.392424Z:517:cc-oci-runtime:debug:sending message (length 349) to proxy socket
2017-02-16T15:05:46.392462Z:517:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hyper","data":{"hyperName":"startpod","data":{"hostname":"dfd9317bddbd","containers":[],"shareDir":"rootfs","interfaces":[{"device":"enp0s8","newDeviceName":"eth0","ipAddresses":[{"ipAddress":"172.17.0.2","netMask":"255.255.0.0"}]}],"routes":[{"dest":"default","gateway":"172.17.0.1","device":"eth0"},{"dest":"172.17.0.0","device":"eth0"}]}}}
2017-02-16T15:07:45.738128Z:519:cc-oci-runtime:critical:failed to read proxy socket fd
2017-02-16T15:07:45.742396Z:557:cc-oci-runtime:debug:called as: /usr/bin/cc-oci-runtime delete dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674
2017-02-16T15:07:45.742652Z:557:cc-oci-runtime:debug:unable to parse '/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/config.json'
2017-02-16T15:07:45.742664Z:557:cc-oci-runtime:debug:Error parsing '/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/config.json': Failed to open file '/run/docker/libcontainerd/dfd9317bddbd683ae399f6ed4088ce7a0bb410d3fea4554a43b7b4ea32f37674/config.json': No such file or directory

@gorozco1
Copy link
Contributor

@amsokol Could you also paste the output of

/usr/bin/cc-oci-runtime --version

@amsokol
Copy link
Author

amsokol commented Feb 16, 2017

Dear @gorozco1,

Done:
cc-oci-runtime version: 2.1.0-rc.6
spec version: 1.0.0-rc1
commit: 6ae1ebf

@jodh-intel
Copy link
Contributor

Hi @amsokol - thanks again for reporting this issue.

Unfortunately it is not possible to run a Clear Container in a nested hyperv environment, as documented by Microsoft here:

@amsokol
Copy link
Author

amsokol commented Mar 5, 2017

Dear jodh-intel,

I install trial VMware Workstation 12.5.
I get VM nested virtualization config here:
https://www.rickygao.com.au/blog/enable-hyper-v-under-vmware-workstation-for-your-lab/

I try Clear Linux and Clear Containers for Fedora 25.
COR runtime has failed in both cases.

Does Clear Containers supports VMware nested virtualization?

@mcastelino
Copy link
Contributor

@amsokol This is an issue with MSR virtualization when -cpu host is used.

/usr/src/packages/BUILD/target-i386/kvm.c:1777: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

/cc @anthonyzxu

@jodh-intel
Copy link
Contributor

Hi @amsokol - The simplest approach to get going is to install Clear Containers on a Linux host. We have installation instructions for a number of distrubutions here:

If you choose Clear Linux on the host, you only need to run 2 commands:

# Install Clear Containers
sudo swupd bundle-add containers-basic

# Start a container!
sudo docker run -ti fedora bash

(from https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-ClearLinux.md)

@jcvenegas jcvenegas added this to Issues in 2.1.x Stable May 3, 2017
@egernst egernst added the CC 2.1 label Jul 18, 2017
@grahamwhaley grahamwhaley mentioned this issue Aug 1, 2017
Open
@jovizhangwei
Copy link

jovizhangwei commented Aug 15, 2017 •
edited
Loading

I've seen same error in my local host, is there have any solution to this problem?

2017-08-15T08:37:28.123239Z:19805:cc-oci-runtime:debug:arg: 'local,id=workload9p,path=/usr/local/var/run/cc-oci-runtime/66e94784fb461909920d4a72932a3a2541206fd7bef92e5af79594d0b1410b76/workload,security_model=none'
2017-08-15T08:37:28.123819Z:19803:cc-oci-runtime:debug:child setup successful
2017-08-15T08:39:27.950011Z:19806:cc-oci-runtime:critical:failed to read proxy socket fd

#1058

@grahamwhaley
Copy link

As an update, more info, and fixes, over here for CC3.x:
containers/virtcontainers#343

@jcvenegas jcvenegas changed the title Failed to run container in cor runtime Failed to run container in cor runtime using Hyper-V Aug 21, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
CC 2.1 type:bug
Projects
2.1.x Stable
Issues
Milestone
No milestone
Development

No branches or pull requests
9 participants
@amsokol @grahamwhaley @mcastelino @devimc @egernst @jodh-intel @gorozco1 @jcvenegas @jovizhangwei