Hardware breakpoint does not work

[tandasat]

Summary

Hardware breakpoint (HWBP) does not hit after HyperPlatform is installed. After VmInitialization() returned successfully, I set HWBP onto a guest kernel code running on VMware through Windbg from a host. Contents of DR registers seemed fine, but HWBP did not trigger, and also, even if I set HWBP before VmInitialization(), the HWBP did not hit after VmInitialization() was executed.

Workaround

Disable Move-to-DR-exiting by setting false to mov_dr_exiting. This lets HWBP function as it should be.

Related Resources

There are some bug reports with regard to HWBP support on VirtualBox and KVM. Those may or may not be related.

• Hardware breakpoints don't work in Virtualbox (software virtualization only)
  https://www.virtualbox.org/ticket/477
  Not much into to see.
• KVM: x86: fix root cause for missed hardware breakpoints
  https://patchwork.kernel.org/patch/8436141/
• KVM: x86: fix missed hardware breakpoints
  https://patchwork.kernel.org/patch/8359491/
• KVM: Fix and improve guest debugging and x86 debug registers
  http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22838
• kvm-userspace: Fix and improve guest debugging and x86 debug registers
  http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22818

Steps to Reproduce

Disassembly used for below test

$$
$$ break at the entry point of the driver
$$
MemoryMon!DriverEntry+0x9b:
fffff801`1f15409b int     3

$$ 
$$ Set software BP (SWBP) and HWBP to verify HWBP works yet
$$
kd> bp fffff801`1f1540a1
kd> ba e 1 fffff801`1f1540a6 
kd> g
Breakpoint 0 hit
MemoryMon!DriverEntry+0xa1:
fffff801`1f1540a1 call    MemoryMon!ExInitializeDriverRuntime (fffff801`1f1315e0)

$$
$$ The SWBP hit. Check contents of DR registers
$$ => DR0 seems good
$$
kd> rM 20
dr0=fffff8011f1540a6 dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001506f8
kdr0=fffff8011f1540a6 kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401
MemoryMon!DriverEntry+0xa1:
fffff801`1f1540a1 call    MemoryMon!ExInitializeDriverRuntime (fffff801`1f1315e0)

$$
$$ The HWBP works yet
$$
kd> g
Breakpoint 1 hit
MemoryMon!DriverEntry+0xa6:
fffff801`1f1540a6 mov     byte ptr [rsp+20h],0

$$
$$ Clear old BPs. Set SWBP and HWBP on code after VmInitialization()
$$
kd> bc *
kd> bp fffff801`1f154206
kd> ba e 1 fffff801`1f15420b 

$$
$$ Hit the SWBP. 
$$
kd> g
Breakpoint 0 hit
MemoryMon!DriverEntry+0x206:
fffff801`1f154206 cmp     dword ptr [rsp+24h],0

$$
$$ Contents of DR registers seem good
$$
kd> rM 20
dr0=fffff8011f15420b dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001526f8
kdr0=fffff8011f15420b kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401
MemoryMon!DriverEntry+0x206:
fffff801`1f154206 cmp     dword ptr [rsp+24h],0 ss:0018:ffffd001`321f7924=00000000

$$
$$ HWBP at fffff8011f15420b did not fire for some reasons, while DR registers still seem fine
$$
kd> g
MemoryMon!TestRwe+0x37:
fffff801`1f13bb07 int     3
kd> rM 20
dr0=fffff8011f15420b dr1=0000000000000000 dr2=0000000000000000
dr3=0000000000000000 dr6=00000000ffff0ff0 dr7=0000000000000401 cr4=00000000001526f8
kdr0=fffff8011f15420b kdr1=0000000000000000 kdr2=0000000000000000
kdr3=0000000000000000 kdr6=00000000ffff0ff0 kdr7=0000000000000401

[fadetrack]

I get a way from Xen to make Hardware breakpoint (HWBP) work.
But I don't know what happend and it looked like that the way didn't work.
Using this way ,you could find something interesting.
And If you set DRX_MAGICNUM more than 5 (if test in Multi processor, cant't set DRX_MAGICNUM more than 1),it also can't work.
This is the code I added for a test.
If you find the reason or the solution,please tell me.THANK YOU!

Steps:

see attachment. It is vmm.cpp .

vmm.cpp.txt

Sorry, I am not OK in English.

test in win7 x86 sp1 no debug and just 1 processer

let DRX_MAGICNUM <= 5 ,it's look like works properly.

let DRX_MAGICNUM > 5, it's failed.

I think this problem may be caused by read from Drx.

[tandasat]

Thank you for analyzing the issue! Those are interesting findings. Looks like DR0 does not retain its value for some reasons.

It seems the attached code basically limits a number of VM-exit via DR access to DRX_MAGICNUM per one process execution context, is that right? I do not see why it works but definitely a good start point for further analysis.

Can you possibly share code of testHWBP? I do not have any test code setting HWBP now, so I am thinking that it is going to be faster.

[fadetrack]

Yes, I tried to limit the number of Dr-Access when switching the process context.Although I don't know whether it make sense to limit the times, but it is effective.
testHWBP.cpp.txt

[asamy]

Might want to look at saving debug controls in vm exit / vm entry as well, not sure if it relates.

[tandasat]

@asamy Thanks a lot for your help, that was it.

@fadetrack This should be fixed with 96dd681. Can you please check again? Your test code was very helpful.