PT initialization fails when mapping PT buffer

[v-p-b]

xenforeignmemory_map_resource fails (returns 0) when initializing PT:

https://github.com/intel/kernel-fuzzer-for-xen-project/blob/master/src/ptcov.c#L113

Any tips on what can be the cause / how to debug this?

[tklengyel]

That happens if you didn't start the parent VM with the pt buffer size specified (ie processor_trace_buf_kb=65536).

[v-p-b]

I have that line in the .cfg file used when creating the domain. I installed the OS earlier without that setting though, can that be a problem? Does the position of this line matter in the config? When calling xenforeignmemory_map_resource, pt_buf_size also seems to be properly set, although I don't know if that's related to the VMs config - my military grade debug print shows something like:

xenforeignmemory_map_resource(0x0x563a887213c0,4,2,0,0,0x4000,0x0x563a840c7898,PROT_READ,0); 

[tklengyel]

I installed the OS earlier without that setting though, can that be a problem? Does the position of this line matter in the config?

No, you only need to add that when you need it, it's not exposed to the guest. It doesn't matter where in the config you have it either. The only time I ran into that was when I booted the VM without the option set. There is a tool in the Xen folder in tools/proctrace, try to compile that and see if you can get it working on your VM. That would help narrow down where the issue might be.

[v-p-b]

Same result with proctrace:

root@zero1:/home/b/debian# xl create debian.cfg 
Parsing config from debian.cfg
root@zero1:/home/b/debian# cat debian.cfg 
name="debian"
builder="hvm"
vcpus=1
maxvcpus=1
memory=1024
maxmem=1024
hap=1
boot="cd"
serial="pty"
vif=['bridge=xenbr0']
vnc=1
vnclisten="0.0.0.0"
vncpasswd='1234567'
usb=1
usbdevice=['tablet']
processor_trace_buf_kb=65536
# Make sure to update the paths below!
disk=['file:/home/b/debian/vmdisk.img,xvda,w',
      ]
root@zero1:/home/b/debian# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0  4094     2     r-----      18.4
debian                                       2  1016     1     -b----       6.8
root@zero1:/home/b/debian# ../kfx-pt/xen/tools/proctrace/proctrace 2 0
Failed to map trace buffer

Is there maybe some BIOS setting I have to activate for PT (I had to manually enable VT-x for example)? Here's how one of my cores looks (running on bare hw):

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Core(TM) i7-9700F CPU @ 3.00GHz
stepping	: 13
microcode	: 0xca
cpu MHz		: 800.441
cache size	: 12288 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit
bogomips	: 6000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

[tklengyel]

Does xl dmesg show anything relevant perhaps? I didn't have to set anything in the bios on any of the systems I tried so far to get it enabled, just vt-x. I'm afraid you might need to go into Xen and add some printk's to try to figure out why that hypercall might fail. Maybe Michal who developed the Xen PT patches has some clue (/cc @icedevml)?

[icedevml]

will look tomorrow

[icedevml]

@tklengyel I think you are missing Andrew Cooper's acquire_resource patches, could you rebase xen to:
https://github.com/icedevml/xen/commits/ipt-patch-v7s ?

It contains a whole lot of patches from Andrew that are fixing things around acquire_resource in particular.

[tklengyel]

@icedevml Yea, it's been on my TODO to move the submodule up closer to the latest version of the PT series. @v-p-b you can try Xen from that branch in the interim!

[v-p-b]

I tried the ipt-patch-v7s tag, but no luck :(

I see this in xl dmesg:

(d2) Booting from Hard Disk...
(d2) Booting from 0000:7c00
(XEN) stdvga.c:178:d2v0 leaving stdvga mode
(XEN) stdvga.c:173:d2v0 entering stdvga mode
(XEN) Dom2 callback via changed to Direct Vector 0xf3
(XEN) irq.c:378: Dom2 PCI link 0 changed 5 -> 0
(XEN) irq.c:378: Dom2 PCI link 1 changed 10 -> 0
(XEN) irq.c:378: Dom2 PCI link 2 changed 11 -> 0
(XEN) irq.c:378: Dom2 PCI link 3 changed 5 -> 0
(XEN) vmx.c:3184:d2v0 RDMSR 0x0000064e unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000034 unimplemented
(XEN) grant_table.c:1841:d2v0 Expanding d2 grant table from 1 to 2 frames
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000606 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000611 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000639 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000641 unimplemented
(XEN) vmx.c:3184:d2v0 RDMSR 0x00000619 unimplemented

I'm attaching the full dmesg log:
dmesg_d3.txt

[v-p-b]

OK, I just realized, that my kernel doesn't have PT support somehow (/sys/devices/intel_pt doesn't exists). I don't have a clue what can cause this, but I keep digging, and close this issue for now.

[v-p-b]

After a kernel upgrade I can use PT with perf on bare HW but not from dom0 (I recompiled Xen and booting the new kernel). Is this normal?

bare metal:

root@zero1:/home/b# perf record -e intel_pt// id
uid=0(root) gid=0(root) groups=0(root)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.120 MB perf.data ]
root@zero1:/home/b# uname -a
Linux zero1 4.15.0-124-generic #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@zero1:/home/b# xen-detect
Not running on Xen.

dom0:

root@zero1:/home/b# xen-detect 
Running in PV context on Xen V4.15.
root@zero1:/home/b# uname -a
Linux zero1 4.15.0-124-generic #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@zero1:/home/b# perf record -e intel_pt// id
event syntax error: 'intel_pt//'
                     \___ Cannot find PMU `intel_pt'. Missing kernel support?
Run 'perf list' for a list of valid events

 Usage: perf record [<options>] [<command>]
    or: perf record [<options>] -- <command> [<options>]

    -e, --event <event>   event selector. use 'perf list' to list available events

[icedevml]

I think this is normal. Xen masks some features in such way that they couldn't be normally used.

[icedevml]

I think you've ran into some bug around acquire_resource. I will look at this soon.

[v-p-b]

I added some debug strings:

diff --git a/tools/libs/foreignmemory/linux.c b/tools/libs/foreignmemory/linux.c                                                                                                                                                              
index eec089e..4fd6ae9 100644
--- a/tools/libs/foreignmemory/linux.c
+++ b/tools/libs/foreignmemory/linux.c
@@ -317,8 +317,10 @@ int osdep_xenforeignmemory_map_resource(
 
     fres->addr = mmap(fres->addr, fres->nr_frames << PAGE_SHIFT,
                       fres->prot, fres->flags | MAP_SHARED, fmem->fd, 0);
-    if ( fres->addr == MAP_FAILED )
-        return -1;
+    if ( fres->addr == MAP_FAILED ){
+      PERROR("mmap failed"); 
+      return -1;
+    }
 
     mr.addr = (uintptr_t)fres->addr;
 
@@ -327,11 +329,13 @@ int osdep_xenforeignmemory_map_resource(
     {
         int saved_errno;
 
-        if ( errno != fmem->unimpl_errno && errno != EOPNOTSUPP )
+        if ( errno != fmem->unimpl_errno && errno != EOPNOTSUPP ){
             PERROR("ioctl failed");
-        else
+       }
+        else{
             errno = EOPNOTSUPP;
-
+           PERROR("EOPNOTSUPP");
+       }
         saved_errno = errno;
         (void)osdep_xenforeignmemory_unmap_resource(fmem, fres);
         errno = saved_errno;

Got this when creating the domain:

# xl create debianpt.cfg                                                                                                                                                                                             
Parsing config from debianpt.cfg                                                                                                                                                                                                              
xenforeignmemory: error: EOPNOTSUPP: Operation not supported


# ./proctrace 1 0
xenforeignmemory: error: EOPNOTSUPP: Operation not supported
Failed to map trace buffer

So it seems IOCTL_PRIVCMD_MMAP_RESOURCE is failing, but unfortunately I'm lost at this point about where this IOCTL is supposed to be handled and I can't debug further.

[tklengyel]

The ioctl is handled by the xen_privcmd kernel module, which is effectively just a bouncer to forward the syscall as a hypercall to Xen. I find it interesting that you get an EOPNOTSUPP error though. To me that sounds like you may have a mismatch between the toolstack and the running version of Xen - perhaps the toolstack has the PT functions defined but the version of Xen you are running doesn't understand the hypercalls? Can you make sure you are running the version of Xen that has the PT patches? You could perhaps also add a gdprintk to these function in Xen (http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/arch/x86/hvm/vmx/vmx.c;h=446fbdde08a53ce4b4d136c8b3976c4c5c1846e2;hb=refs/heads/ptcov2#l431 and http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/arch/x86/hvm/vmx/vmx.c;h=446fbdde08a53ce4b4d136c8b3976c4c5c1846e2;hb=refs/heads/ptcov2#l2366) to verify that PT does get enabled for the VM.

[v-p-b]

Full-spectrum printk()'s show that the return 0 path is taken in vmx_init_pt():

http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/arch/x86/hvm/vmx/vmx.c;h=446fbdde08a53ce4b4d136c8b3976c4c5c1846e2;hb=refs/heads/ptcov2#l469

(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 556kB init memory
(XEN) d0: Forcing write emulation on MFNs e0000-effff
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:17.0
(XEN) PCI add device 0000:00:1b.0
(XEN) PCI add device 0000:00:1c.0
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:00:1f.4
(XEN) PCI add device 0000:00:1f.5
(XEN) PCI add device 0000:00:1f.6
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:01:00.1
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x0000064e unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000034 unimplemented
(XEN) d0: Forcing read-only access to MFN fed00
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000606 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000606 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x0000060d unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003f8 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003f9 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x000003fa unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000630 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000631 unimplemented
(XEN) emul-priv-op.c:991:d0v1 RDMSR 0x00000632 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x0000060d unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003f8 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003f9 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x000003fa unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000630 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000631 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000632 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000611 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000639 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000641 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000619 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000611 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000639 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000641 unimplemented
(XEN) emul-priv-op.c:991:d0v0 RDMSR 0x00000619 unimplemented
(XEN) vmx.c:476:d0v1 XXXIPT return 0 from vmx_init_pt<G><2>HVM d1v0 save: CPU

I noticed the unimplemented MSR's before, but they looked to be related to energy consumption sensors (the current list is longer than I remember though). I see MSR_RTIT_CTL referenced by vmx_init_pt() defined as 0x570, not reported above (or anywhere else in xl dmesg output).

[tklengyel]

So it seems your hardware is fine and PT gets setup correctly. Just have to figure out why mapping the pt buffer into dom0 fails for you. The XENMEM_resource_vmtrace_buf hypercall should eventually reach http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=xen/common/memory.c;h=c0a22eb60f698b1a1dc45027adf7b3a5ed73a1d2;hb=refs/heads/ptcov2#l1010. Can you check if that's reached and what it returns?

[tklengyel]

Interestingly I don't see EOPNOTSUPP being set for this call anywhere O.o

[tklengyel]

Never mind, found it, it's at http://xenbits.xen.org/gitweb/?p=people/tklengyel/xen.git;a=blob;f=tools/libs/foreignmemory/linux.c;h=8daa5828e3da2f1d0dee158ff91e8ba2274a4625;hb=refs/heads/ptcov2#l328. So yea, let's check what the hypervisor is doing when this request reaches it, I doubt the issue would be on the linux kernel module side that handles the ioctl.

[icedevml]

Sorry for not looking at it, backlog overflow. Maybe this is hitting some blacklist/whitelist on the Xen side? I've already seen such thing twice, there are some code pieces that are performing numerous sanity checks.

[v-p-b]

I'm still debugging this, but so far I wasn't able to trigger any gdprintk's in or on the way to acquire_vmtrace_buf.

Meanwhile can you tell me which kernel modules I should see in dom0? I can see some loading failures during boot, but I don't know what exactly is failing yet (systemd ftw!). Here's what I see:

# lsmod | fgrep xen
xen_pciback            61440  0
xen_netback            57344  1
xen_blkback            45056  1
xen_gntalloc           16384  0
xenfs                  16384  1
xen_gntdev             20480  1
xen_privcmd            20480  39 xenfs
xen_evtchn             16384  5

[v-p-b]

I found that do_memory_op() (in xen/common/memory.c) is the main dispatcher that should lead execution to acquire_resource (or am I missing something?). I placed debug prints in there, and found that the XENMEM_acquire_resource branch is never called. I also printed all operation leading to the default branch of the switch (decimals at the end of lines are op codes):

# xl dmesg | fgrep XXX
(XEN) memory.c:1702:d0v0 XXXIPT do_memory_op default path12
(XEN) memory.c:1702:d0v0 XXXIPT do_memory_op default path10
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path18
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path19
(XEN) memory.c:1702:d0v1 XXXIPT do_memory_op default path13

XENMEM_acquire_resource is defined as 28. So it seems that the appropriate operation is never requested.

Interestingly, the number of debug messages doesn't grow as I run the proctrace tool against the domain (dom1 in this example).

Any tips/help on this is appreciated!

[tklengyel]

Meanwhile can you tell me which kernel modules I should see in dom0?

The ones you have look fine, really just the default list, no special kernel module is needed.

XENMEM_acquire_resource is defined as 28. So it seems that the appropriate operation is never requested.

That's certainly a weird observation. That would mean the map resource from the dom0 userpsace never actually reaches Xen O.o

[tklengyel]

@v-p-b I would certainly suggest trying it on another machine as well as this issue hasn't popped up on anything I tried so far.

[v-p-b]

I don't have a spare machine that supports PT unfortunately :(

Can you give me some high level description/pointers about how memory operations should reach Xen, so I can narrow down where the signal gets lost?

[v-p-b]

OK, this is a bit embarrassing: a quick chat with @tklengyel revealed that I was using an outdated/unsupported Ubuntu version, where the kernel didn't have support for IOCTL_PRIVCMD_MMAP_RESOURCE. Upgrading to 20.04 (5.4 kernel) resolved the issue.