New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QEMU/haxm: Multiple segmentation faults on Ubuntu 18.04 #39

Closed
mifritscher opened this Issue Apr 4, 2018 · 13 comments

Comments

Projects
None yet
3 participants
@mifritscher
Copy link

mifritscher commented Apr 4, 2018

While installation of a Ubuntu 16.04 works fine with qemu 2.11 and haxm 7.1, 18.04 produces multiple user mode segmentation faults in the guest on early steps and eventually hangs. The netboot images can be obtained from http://de.archive.ubuntu.com/ubuntu/dists/artful/main/installer-amd64/current/images/netboot/netboot.tar.gz .

Parameters:
-m 1024 -smp 2 -rtc base=utc -drive file=temp.vmdk,if=virtio,detect-zeroes=unmap,discard=unmap -drive file=fat:rw:fat-type=32:label=kernel:C:\temp,format=raw,if=virtio -vga std -device virtio-net,netdev=natted -netdev user,id=natted -kernel linux_initrd/linux -initrd linux_initrd/initrd.gz -append "nofb fb=false pti=off interface=auto auto=true" -nodefaults

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 4, 2018

Seems to be some sort of race condition. If I remove the SMP option the failure sqeuence is a bit different, but it still crashes at the early beginning.

@raphaelning

This comment has been minimized.

Copy link
Contributor

raphaelning commented Apr 9, 2018

Thanks for the bug report, as well as for all the other issues you've filed! We are busy with something else at the moment, but we'll start looking into these issues later this week.

@raphaelning raphaelning added the bug label Apr 9, 2018

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 9, 2018

Ok, thanks for your update!

@raphaelning

This comment has been minimized.

Copy link
Contributor

raphaelning commented Apr 13, 2018

Our QA has reproduced this issue, but we need more details about the segmentation fault, e.g. a core dump. Does the Ubuntu installer do that? Does it keep an installation log somewhere?

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 13, 2018

On alt+F4 you get the current log. I think it writes a install log onto /var/log as well.
Regarding core dumps: I think(!) the installer makes core dumps somewhere - either in /, /tmp or /var (using the normal linux core dump facility).

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 20, 2018

You could try the hints shown in https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/ immediately after start (switch to alt-f2)

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 20, 2018

Further insights: right after the message box, make a alt+f2, enter, find | grep /core . Sometimes there are error messages like "%s%s%s not found". Sometimes (seldom) it even crashes.

Current bootline:

C:\temp\_haxm_qemu>qemu-system-x86_64.exe -m 1024 -smp 2 -drive file=fat:rw:fat-type:C:\temp\_haxm_qemu\_img  -kernel linux  -initrd initrd.gz -append "nofb fb=false pti=off" -enable-hax -serial file:
test

I run:

ulimit -S -c unlimited
while [ true ]; do
find >/dev/zero
done

After a while (30 seconds to 2 minutes), I got a segmentation fault. I wrote the coredump with dd if=core of=/dev/ttyS0 outside of the VM.

I packeted following things into https://mifritscher.de/austausch/_packet.zip :

  • linux/initrd
  • coredump
  • dmesg

I get the same symptoms also with only -smp 1 .

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Apr 30, 2018

Do you need any additional information?

@raphaelning

This comment has been minimized.

Copy link
Contributor

raphaelning commented May 2, 2018

Sorry for the late response, and thanks a lot for providing the additional information. I tried to load the core dump (_packet/test) into gdb but it wouldn't recognize the format :(

$ gdb -c test
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
...
"<redacted>/test" is not a core dump: File format not recognized

Anything I did wrong?

Meanwhile, the latest Android 4.9 kernel also runs into segfaults under HAXM:

https://issuetracker.google.com/issues/78465772
(long thread, but you get the point if you search for SIGSEGV)

I'm not sure if these two issues share the same root cause. But at least Android dumps more information about the crash, so maybe we should look into the Android issue first (and kill two birds with one stone, hopefully).

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented May 2, 2018

Grml - ok, it seems that the file somehow got \r inserted... I hackfixed it and re-uploaded the file under https://mifritscher.de/austausch/test.core. Then, gdb can at least read the file:

[New LWP 475]
Core was generated by `find'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()

(gdb) info target
Local core dump file:
	`/tmp/test.core', file type elf64-x86-64.
	0x000055b2d2267000 - 0x000055b2d2268000 is load1a
	0x000055b2d2268000 - 0x000055b2d2268000 is load1b
	0x000055b2d24b4000 - 0x000055b2d24b6000 is load2
	0x000055b2d24b6000 - 0x000055b2d24b7000 is load3
	0x000055b2d3280000 - 0x000055b2d32a1000 is load4
	0x00007f45cf315000 - 0x00007f45cf316000 is load5a
	0x00007f45cf316000 - 0x00007f45cf316000 is load5b
	0x00007f45cf4fc000 - 0x00007f45cf4fc000 is load6
	0x00007f45cf6fc000 - 0x00007f45cf700000 is load7
	0x00007f45cf700000 - 0x00007f45cf702000 is load8
	0x00007f45cf702000 - 0x00007f45cf706000 is load9
	0x00007f45cf706000 - 0x00007f45cf707000 is load10a
	0x00007f45cf707000 - 0x00007f45cf707000 is load10b
	0x00007f45cf8da000 - 0x00007f45cf92d000 is load11
	0x00007f45cf92d000 - 0x00007f45cf92e000 is load12
	0x00007f45cf92e000 - 0x00007f45cf92f000 is load13
	0x00007f45cf92f000 - 0x00007f45cf930000 is load14
	0x00007ffdb921f000 - 0x00007ffdb9240000 is load15
	0x00007ffdb92db000 - 0x00007ffdb92de000 is load16
	0x00007ffdb92de000 - 0x00007ffdb92e0000 is load17
	0xffffffffff600000 - 0xffffffffff601000 is load18


(gdb) info frame
Stack level 0, frame at 0x7ffdb923e5f0:
 rip = 0x0; saved rip = 0x7f45cf3a4fe7
 called by frame at 0x7ffdb923e5f8
 Arglist at 0x7ffdb923e5e0, args: 
 Locals at 0x7ffdb923e5e0, Previous frame's sp is 0x7ffdb923e5f0
 Saved registers:
  rip at 0x7ffdb923e5e8

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f45cf3a4fe7 in ?? ()
#2  0x000055b2d329a190 in ?? ()
#3  0x000055b2d329a190 in ?? ()
#4  0x000055b2d3299cb0 in ?? ()
#5  0x000055b2d3299cb0 in ?? ()
#6  0x00007ffdb923e6e0 in ?? ()
#7  0x00007f45cf8da0bb in ?? ()
#8  0x0000000000000d68 in ?? ()
#9  0x00007f45cf6fc760 in ?? ()
#10 0x0000000000000014 in ?? ()
#11 0x0000000000000005 in ?? ()
#12 0x00007ffdb923ec20 in ?? ()
#13 0x00007f45cf3a3494 in ?? ()
#14 0x000055b2d3299d90 in ?? ()
#15 0x0000003500000014 in ?? ()
#16 0x0000000000000000 in ?? ()

(gdb) info all-registers
rax            0x64	100
rbx            0x7ffdb923ec20	140727709592608
rcx            0x0	0
rdx            0x55b2d3299d14	94226535259412
rsi            0x64	100
rdi            0x12c	300
rbp            0x6d	0x6d
rsp            0x7ffdb923e5e8	0x7ffdb923e5e8
r8             0x7f45cf8da0ab	139937811636395
r9             0x14	20
r10            0xffffffec	4294967276
r11            0x55b2d22b32b5	94226518586037
r12            0x55b2d3299cb0	94226535259312
r13            0x14	20
r14            0x12c	300
r15            0x64	100
rip            0x0	0x0
eflags         0x10293	[ CF AF SF IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
st0            0	(raw 0x00000000000000000000)
st1            0	(raw 0x00000000000000000000)
st2            0	(raw 0x00000000000000000000)
st3            0	(raw 0x00000000000000000000)
st4            0	(raw 0x00000000000000000000)
st5            0	(raw 0x00000000000000000000)
st6            0	(raw 0x00000000000000000000)
st7            0	(raw 0x00000000000000000000)
fctrl          0x37f	895
fstat          0x0	0
ftag           0xffff	65535
fiseg          0x0	0
fioff          0x0	0
foseg          0x0	0
fooff          0x0	0
fop            0x0	0
xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x8000000000000000}, v16_int8 = {0x0 <repeats 15 times>, 0xff}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff00}, v4_int32 = {0x0, 0x0, 0x0, 0xff000000}, 
  v2_int64 = {0x0, 0xff00000000000000}, uint128 = 0xff000000000000000000000000000000}
xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0x0, 0x0, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xff, 0xff}, v8_int16 = {0x0, 
    0xffff, 0xffff, 0xffff, 0x0, 0x0, 0xffff, 0xffff}, v4_int32 = {0xffff0000, 0xffffffff, 0x0, 0xffffffff}, v2_int64 = {0xffffffffffff0000, 0xffffffff00000000}, uint128 = 0xffffffff00000000ffffffffffff0000}
xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0, 0xff, 0x0 <repeats 14 times>}, v8_int16 = {0xff00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0xff00, 0x0, 0x0, 0x0}, v2_int64 = {
    0xff00, 0x0}, uint128 = 0xff00}
xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x67, 0x65, 0x74, 0x31, 0x3a, 0x30, 0x3a, 0x30, 0x2f, 0x31, 0x3a, 0x30, 0x3a, 0x30, 0x3a, 0x30}, v8_int16 = {0x6567, 0x3174, 0x303a, 0x303a, 
    0x312f, 0x303a, 0x303a, 0x303a}, v4_int32 = {0x31746567, 0x303a303a, 0x303a312f, 0x303a303a}, v2_int64 = {0x303a303a31746567, 0x303a303a303a312f}, uint128 = 0x303a303a303a312f303a303a31746567}
xmm6           {v4_float = {0xffffffff, 0xffffffff, 0x0, 0xffffffff}, v2_double = {0x7fffffffffffffff, 0x7fffffffffffffff}, v16_int8 = {0x31, 0x2f, 0x61, 0x74, 0x61, 0x32, 0x2f, 0x68, 0x6f, 0x73, 0x74, 0x31, 0x2f, 0x74, 0x61, 
    0x72}, v8_int16 = {0x2f31, 0x7461, 0x3261, 0x682f, 0x736f, 0x3174, 0x742f, 0x7261}, v4_int32 = {0x74612f31, 0x682f3261, 0x3174736f, 0x7261742f}, v2_int64 = {0x682f326174612f31, 0x7261742f3174736f}, 
  uint128 = 0x7261742f3174736f682f326174612f31}
xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x30, 0x3a, 0x30, 0x30, 0x2f, 0x30, 0x30, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x3a, 0x30, 0x31, 0x2e}, v8_int16 = {0x3a30, 0x3030, 0x302f, 0x3030, 
    0x3a30, 0x3030, 0x303a, 0x2e31}, v4_int32 = {0x30303a30, 0x3030302f, 0x30303a30, 0x2e31303a}, v2_int64 = {0x3030302f30303a30, 0x2e31303a30303a30}, uint128 = 0x2e31303a30303a303030302f30303a30}
xmm8           {v4_float = {0xfebe000, 0x0, 0xfebe000, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x20, 0x14, 0x70, 0xcf, 0x45, 0x7f, 0x0, 0x0, 0x20, 0x14, 0x70, 0xcf, 0x45, 0x7f, 0x0, 0x0}, v8_int16 = {0x1420, 0xcf70, 0x7f45, 
    0x0, 0x1420, 0xcf70, 0x7f45, 0x0}, v4_int32 = {0xcf701420, 0x7f45, 0xcf701420, 0x7f45}, v2_int64 = {0x7f45cf701420, 0x7f45cf701420}, uint128 = 0x7f45cf70142000007f45cf701420}
xmm9           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm10          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm11          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm12          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm13          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
---Type <return> to continue, or q <return> to quit---
xmm14          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
xmm15          {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, 
  uint128 = 0x0}
mxcsr          0x1f80	[ IM DM ZM OM UM PM ]

Sadly, the backtrace isn't such usefull, but the binaries are in the initramdisk.

@mifritscher

This comment has been minimized.

Copy link

mifritscher commented May 2, 2018

Regarding android: Yes, that could be an idea ;-)

@junxiaoc

This comment has been minimized.

Copy link
Contributor

junxiaoc commented Jul 31, 2018

This issue looks similar with #74 . We are working on fix.

junxiaoc added a commit that referenced this issue Jul 31, 2018

core: save/restore FPU registers in VM entry/exit
Guest OS kernel/app might use SSE instruction and registers.
When Guest OS VMM exits, these registers should be saved, or else
it might be corrupted by host OS/app. In next time guest VMM
enter, guest's SSE registers context might be corrupted.

Guest app segment fault, coredump, and kernel panic were reported
which should be related with this issue.

This change is to remove is_fpu_used flag so guest FPU registers
could be saved in VM exit and restored in VM enter unconditionally.

Fixes #39, fixes #74.

junxiaoc added a commit that referenced this issue Aug 1, 2018

core: save/restore FPU registers in VM entry/exit
Guest OS kernel/app might use SSE instruction and registers.
When Guest OS VMM exits, these registers should be saved, or else
it might be corrupted by host OS/app. In next time VM entry,
guest's SSE registers context might be corrupted.

Guest app segfault and kernel panic were reported which should
be related with this issue.

This change is to remove is_fpu_used flag so guest FPU registers
could be saved in VM exit and restored in VM entry unconditionally.

Fixes #39, fixes #74.

junxiaoc added a commit that referenced this issue Aug 1, 2018

core: save/restore FPU registers in VM entry/exit
Guest OS kernel/app might use SSE instruction and registers.
When Guest OS VM exits, these registers should be saved, or else
it might be corrupted by host OS/app. In next time VM entry,
guest's SSE registers context might be corrupted.

Guest app segfault and kernel panic were reported which should
be related with this issue.

This change is to remove is_fpu_used flag so guest FPU registers
could be saved in VM exit and restored in VM entry unconditionally.

Fixes #39, fixes #74.

junxiaoc added a commit that referenced this issue Aug 1, 2018

core: save/restore FPU registers in VM entry/exit
Guest OS kernel/app might use SSE instruction and registers.
When Guest OS VM exits, these registers should be saved, or else
it might be corrupted by host OS/app. In next time VM entry,
guest's SSE registers context might be corrupted.

Guest app segfault and kernel panic were reported which should
be related with this issue.

This change is to remove is_fpu_used flag so guest FPU registers
could be saved in VM exit and restored in VM entry unconditionally.

Fixes #39, fixes #74.
@mifritscher

This comment has been minimized.

Copy link

mifritscher commented Oct 6, 2018

I can confirm that this issue is fixed with 7.3.2! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment