sudden VM restarts #1305

errordeveloper · 2017-02-15T07:34:38Z

Expected behavior

Stable operation.

Actual behavior

A crash.

The logs attached to diagnostic 722655D0-C8CB-40C5-AF4C-1964C426C922 should contain info about at least 2 incidents of sudden VM restarts (daemon crashes?).

Information

Full output of the diagnostics from "Diagnose & Feedback" in the menu

Docker for Mac: version: 1.13.1 (94675c5a7)
macOS: version 10.12.3 (build: 16D32)
logs: /tmp/722655D0-C8CB-40C5-AF4C-1964C426C922/20170215-054548.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Diagnostic ID: 722655D0-C8CB-40C5-AF4C-1964C426C922

Steps to reproduce the behavior

I'm still working through this, it's a bit complicated and I don't want to provide potentially misleading info. What I am trying to do is run Kubernetes on Docker for Mac (with kubelet in a privileged container), and it seems to make Docker for Mac crash.

The last crash was yesterday in the afternoon.

Please take a look at grep '2017-02-14 16:35' docker-system.log, it seems to reveal some details:

2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: vm exit[2]
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	reason		VMX
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	rip		0xffffffff81841afd
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	inst_length	5
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	status		0
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	exit_reason	2
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	qualification	0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	inst_type		0
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 	inst_error		0
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: virtio-net-vpnkit: initialising, opts="uuid=d363d91b-7950-4ca5-ac5e-11421efd8170,path=/Users/ilya/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/ilya/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0"
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: Interface will have uuid d363d91b-7950-4ca5-ac5e-11421efd8170
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: Connection established with MAC=c0:ff:ee:c0:ff:ee and MTU 1500
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: virtio-9p: initialising path=/Users/ilya/Library/Containers/com.docker.docker/Data/s40,tag=db
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: virtio-9p: initialising path=/Users/ilya/Library/Containers/com.docker.docker/Data/s51,tag=port
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: linkname /Users/ilya/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: COM1 connected to /dev/ttys005
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: COM1 linked to /Users/ilya/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_PIN_BASED_CTLS:           0x000000000000003f
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_PRI_PROC_BASED_CTLS:      0x00000000b5186dfa
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_SEC_PROC_BASED_CTLS:      0x00000000000000aa
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_ENTRY_CTLS:               0x00000000000093ff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_EXCEPTION_BITMAP:         0x0000000000040000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_CR0_MASK:                 0x00000000e0000031
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_CR0_SHADOW:               0x0000000080050033
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_CR4_MASK:                 0x0000000000002000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_CR4_SHADOW:               0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CS_SELECTOR:        0x0000000000000010
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CS_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: Hypervisor subprocess has shutdown
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CS_ACCESS_RIGHTS:   0x000000000000a09b
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CS_BASE:            0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_DS_SELECTOR:        0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_DS_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_DS_ACCESS_RIGHTS:   0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_DS_BASE:            0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_ES_SELECTOR:        0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_ES_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_ES_ACCESS_RIGHTS:   0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_ES_BASE:            0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_FS_SELECTOR:        0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_FS_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_FS_ACCESS_RIGHTS:   0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_FS_BASE:            0x00007f6e39d96ab0
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GS_SELECTOR:        0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GS_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GS_ACCESS_RIGHTS:   0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GS_BASE:            0xffff8801bfd00000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_SS_SELECTOR:        0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_SS_LIMIT:           0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_SS_ACCESS_RIGHTS:   0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_SS_BASE:            0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_LDTR_SELECTOR:      0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_LDTR_LIMIT:         0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_LDTR_ACCESS_RIGHTS: 0x000000000001c000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_LDTR_BASE:          0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_TR_SELECTOR:        0x0000000000000040
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_TR_LIMIT:           0x0000000000002087
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_TR_ACCESS_RIGHTS:   0x000000000000008b
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_TR_BASE:            0xffff8801bfd16080
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GDTR_LIMIT:         0x000000000000007f
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_GDTR_BASE:          0xffff8801bfd09000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_IDTR_LIMIT:         0x00000000ffffffff
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_IDTR_BASE:          0x0000000000000000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CR0:                0x0000000080050033
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CR3:                0x000000016b036000
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_CR4:                0x00000000000426a0
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: VMCS_GUEST_IA32_EFER:          0x0000000000000d01
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: 
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: rip: 0xffffffff81841afd rfl: 0x0000000000010087 cr2: 0xffff8801bfd04ff8
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: rax: 0x0000000081841157 rbx: 0x0000000000000001 rcx: 0xffffffff81841157 rdx: 0x0000560b55a9b560
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: rsi: 0x0000000000000010 rdi: 0xffffc9000735b038 rbp: 0x0000560b55a9b550 rsp: 0xffffc9000735af88
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: r8:  0x0101010101010101 r9:  0x00000000000005a0 r10: 0x0000000000000000 r11: 0x0000000000000246
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: r12: 0x00007f6e39d934d0 r13: 0x00007f6e39d934d
2017-02-14 16:35:21+00 wroom Docker[com.docker.osxfs][66762] <Error>: Fatal unexpected exception: Socket.Closed
2017-02-14 16:35:21+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Reap com.docker.osxfs (pid 66762): exit status 1
2017-02-14 16:35:21+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: TRIM is enabled but auto-compaction disabled: compacting now
2017-02-14 16:35:21+00 wroom Docker[com.docker.osx.hyperkit.linux][716] <Notice>: VM shutdown at 2017-02-14 16:35:21 +0000 GMT
2017-02-14 16:35:22+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 66764)
2017-02-14 16:35:22+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Signal terminated to com.docker.driver.amd64-linux (pid 66764)
2017-02-14 16:35:22+00 wroom Docker[com.docker.driver.amd64-linux][66764] <Notice>: sending SIGTERM to com.docker.hyperkit pid 66778
2017-02-14 16:35:22+00 wroom Docker[com.docker.osx.hyperkit.linux][716] <Notice>: VM shutdown at 2017-02-14 16:35:22 +0000 GMT
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Reap com.docker.driver.amd64-linux (pid 66764): exit status 0
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Stop 2 children with order 2: com.docker.osxfs (pid 66762), com.docker.slirp (pid 66763)
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Signal terminated to com.docker.slirp (pid 66763)
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Reap com.docker.slirp (pid 66763): signal: terminated
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Starting com.docker.osxfs, com.docker.slirp, com.docker.driver.amd64-linux
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Start com.docker.osxfs (pid 71458)
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Start com.docker.slirp (pid 71459)
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Start com.docker.driver.amd64-linux (pid 71460)
2017-02-14 16:35:32+00 wroom Docker[com.docker.osx.hyperkit.linux][706] <Notice>: Supervisor restarted %d0

The text was updated successfully, but these errors were encountered:

errordeveloper · 2017-02-15T10:09:57Z

~~It occurred again after my Mac went to sleep. So to me it seems as if it's associated with sleep cycles and load, and potentially nothing todo with Kubernetes.~~

errordeveloper · 2017-02-15T11:39:44Z

You should be able to reproduce it with this:

git clone --branch docker-for-mac-issue-1305 https://github.com/errordeveloper/kxd
cd kxd
./reproduce.sh

It'd take a while to pull all the image for the first time, but once you have them and run reproduce.sh again, Docker for Mac crashes pretty much right after this script exists.

errordeveloper · 2017-02-16T09:27:10Z

~~It's possible that #690, #1071 and #1228 are duplicates of this.~~

errordeveloper · 2017-02-16T09:27:38Z

I have switched to beta, and I am able to reproduce it on beta also.

dsheets · 2017-02-16T10:49:54Z

I believe this is a duplicate of #1170 (vm exit code 2, triple fault) but your reproduction needs to be investigated.

errordeveloper · 2017-02-16T12:17:17Z

@dsheets thanks! I'm sorry the reproduction is a bit heavyweight, but it triggers the issue each time on my machine.

ijc · 2017-02-17T13:33:29Z

I've just reproduced using the case given in #1305 (comment), thanks!

In my case I saw the exit code 48 (EPT fault) failure mode which would be #1110 rather than #1170 which is the exit code 2 triple fault failure mode. We are currently tracking them separately but I am increasingly convinced that they are down to the same underlying OSX bug.

I'm going to close this as a dup of #1110 and #1170, please subscribe to both of those for further updates (I expect we will eventually close one of those as a dup of the other too). Thanks for your report and reproducer.

ijc · 2017-02-17T13:37:46Z

Actually #1176 is a better reference than #1110.

errordeveloper · 2017-02-17T15:52:28Z

@ijc25 thanks!

justincormack · 2017-09-12T11:01:02Z

I can't reproduce this on OSX Sierra, 2.8GHz Core i7, MacBook Pro 15inch retina mid 2015, using @errordeveloper script.

docker-robott · 2020-06-22T03:07:55Z

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

errordeveloper added a commit to errordeveloper/kxd that referenced this issue Feb 15, 2017

Make it easy to reproduce docker/for-mac#1305

e5afc38

dave-tucker added osx/10.12.x status/triage version/1.13.1 labels Feb 15, 2017

dsheets added area/hyperkit kind/bug status/acknowledged and removed status/triage labels Feb 16, 2017

ijc closed this as completed Feb 17, 2017

errordeveloper mentioned this issue Apr 2, 2017

Docker consistently crashes when doing anything #1176

Closed

docker locked and limited conversation to collaborators Jun 22, 2020

docker-robott added the lifecycle/locked label Jun 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sudden VM restarts #1305

sudden VM restarts #1305

errordeveloper commented Feb 15, 2017

errordeveloper commented Feb 15, 2017 •

edited

Loading

errordeveloper commented Feb 15, 2017 •

edited

Loading

errordeveloper commented Feb 16, 2017 •

edited

Loading

errordeveloper commented Feb 16, 2017

dsheets commented Feb 16, 2017

errordeveloper commented Feb 16, 2017

ijc commented Feb 17, 2017

ijc commented Feb 17, 2017

errordeveloper commented Feb 17, 2017

justincormack commented Sep 12, 2017

docker-robott commented Jun 22, 2020

sudden VM restarts #1305

sudden VM restarts #1305

Comments

errordeveloper commented Feb 15, 2017

Expected behavior

Actual behavior

Information

Steps to reproduce the behavior

errordeveloper commented Feb 15, 2017 • edited Loading

errordeveloper commented Feb 15, 2017 • edited Loading

errordeveloper commented Feb 16, 2017 • edited Loading

errordeveloper commented Feb 16, 2017

dsheets commented Feb 16, 2017

errordeveloper commented Feb 16, 2017

ijc commented Feb 17, 2017

ijc commented Feb 17, 2017

errordeveloper commented Feb 17, 2017

justincormack commented Sep 12, 2017

docker-robott commented Jun 22, 2020

errordeveloper commented Feb 15, 2017 •

edited

Loading

errordeveloper commented Feb 15, 2017 •

edited

Loading

errordeveloper commented Feb 16, 2017 •

edited

Loading