Debug notes

[dimakuv]

[ This is a dump of random notes. When we collect a critical mass, we will re-organize it as FAQ or blog posts. ]

I decided to put each note in its own "Answer", so it is easier to read and navigate here.

The list of notes:

• R: Out-of-memory failures on some machines
• Get rid of Failed to open Intel SGX device error

[dimakuv]

R: Out-of-memory failures on some machines

Relevant link: gramineproject/examples#40

On 20. September 2022, the periodically tested R workload started failing under the following conditions:

1. a large machine with >100 CPU cores,
2. inside a Docker container, and
3. with CentOS 8.

The surprising part was that the same R workload failed only under this set of conditions -- it didn't fail on the same machine on bare-metal (not in Docker container), it didn't fail on Ubuntu Docker containers, and it didn't fail on small machines (with 4-8 CPU cores).

It turns out that the R workload was updated to version 4.2.1 Patched on 14. September 2022. This was exactly the last date of our previous iteration of weekly R workload's tests.

So this R update explained the conditions 2 and 3. The bare-metal OS on that machine was not updated, so it still had the old version of R. Also, it seems that Ubuntu didn't yet put the updated R package in its repository, but CentOS 8 did. And the Docker image with R was re-built every time before the tests, so it got the updated R package.

So why did the updated R start failing with "out of memory"? This is because something changed in the OpenBLAS and OpenMP dependencies of R. Previously, R (through these dependencies) spawned a small number of threads -- 4 threads were seen in our debug. Now, the updated R by default spawns as many threads as there are CPU cores. And for each thread, 128MB are pre-allocated. Thus, for a large machine with 100 CPU cores, this results in ~13GB of enclave memory required. Whereas our current R manifest specifies only 2GB of enclave size. This explains condition 1.

Solution: fortunately, OpenBLAS + OpenMP can limit the number of threads spawned. Thus, to avoid running out of enclave memory, we simply set OMP_NUM_THREADS environment variable to 4, to limit the number of created threads.

[dimakuv]

Get rid of Failed to open Intel SGX device error

[ Kudos to @anjalirai-intel for helping with this description ]

On some systems, while running Gramine with SGX remote attestation (e.g., running the ra-tls-mbedtls example), users may encounter this error:

[get_driver_type edmm_utility.cpp:116] Failed to open Intel SGX device.
[get_driver_type .../ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:116] Failed to open Intel SGX device.

This error looks scary, but it doesn't "break" Gramine and is benign; you can safely ignore it. What this error means is that one of the Intel SGX DCAP libraries that Gramine depends on tries to create an SGX enclave inside the Gramine SGX enclave (which is impossible, so the DCAP library prints this error and falls back to the "no-SGX-enclave" path).

At a deeper technical level, the following happens. Gramine's RA-TLS feature requires the libsgx_dcap_quoteverify.so library, installed as part of the package libsgx-dcap-ql.

This libsgx-dcap-ql package (developed by the Intel SGX DCAP team) has interesting logic when it is requested to perform DCAP-SGX-quote verification:

1. First tries to perform the "in-enclave verification", by trying to load the QVE (Quote Verification Enclave) enclave and jumping into it for the actual verification logic.
2. If above fails, then performs the "out-of-enclave verification", by doing the actual verification logic in a normal context.

The first step only happens when the libsgx-ae-qve package is also installed on your system. This latter package installs the QVE enclave on the system and makes it available to other libraries/packages.

So currently a proper way to disable the Failed to open Intel SGX device error is to remove the libsgx-ae-qve package from the system (of course, only if you don't need this package for some other SGX runtimes):

sudo apt remove libsgx-ae-qve  # only for Debian/Ubuntu; similar for RHEL/CentOS/AlmaLinux

For more context, see intel/linux-sgx#756.

[dimakuv]

Illegal instruction during lscpu run

When running lscpu tool under gramine-sgx, it fails with illegal instruction:

gramine/CI-Examples/bash$ gramine-sgx bash -c scripts/lscpu_test.sh
...
[P2:T2:lscpu] trace: ---- openat(3, "sys/kernel/osrelease", O_RDONLY, 0000) = -2
[P2:T2:lscpu] trace: ---- stat("/sys/firmware/dmi/tables/DMI", 0x16aae840) = -2
[P2:T2:lscpu] trace: ---- openat(AT_FDCWD, "/sys/firmware/efi/systab", O_RDONLY, 0000) = -2
[P2:T2:lscpu] trace: ---- openat(AT_FDCWD, "/proc/efi/systab", O_RDONLY, 0000) = -2
[P2:T2:lscpu] trace: ---- openat(AT_FDCWD, "/dev/mem", O_RDONLY, 0000) = -2
[P2:T2:lscpu] trace: ---- getuid() = 0x0
warning: Unknown or illegal instruction executed at lscpu+0x7291
[P2:T2:lscpu] debug: Illegal instruction during app execution at 0x16a5d291; delivering to app
[P2:T2:lscpu] debug: killed by signal 4
...

For reproducing, the file scripts/lscpu_test.sh is simply:

#!/bin/bash
lscpu

And the diff in bash.manifest.template:

diff --git a/CI-Examples/bash/manifest.template b/CI-Examples/bash/manifest.template
@@ -7,7 +7,7 @@ libos.entrypoint = "{{ execdir }}/bash"
 loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr{{ arch_libdir }}"
-loader.env.PATH = "{{ execdir }}"
+loader.env.PATH = "{{ execdir }}:/usr/bin"

@@ -15,6 +15,7 @@ fs.mounts = [
   { path = "/usr/lib", uri = "file:/usr/lib" },
+  { path = "/usr/bin/lscpu", uri = "file:/usr/bin/lscpu" },
   { path = "{{ execdir }}", uri = "file:{{ execdir }}" },
@@ -29,6 +30,7 @@ sgx.trusted_files = [
   "file:/usr/{{ arch_libdir }}/",
+  "file:/usr/bin/lscpu",
 ]

Root causing

First, let's understand what instruction we can see at lscpu+0x7291 address:

$ objdump -D /usr/bin/lscpu | grep 7291
    7291:       ed                      in     (%dx),%eax

This makes sense -- the in (Input from Port) x86 instruction is prohibited in Intel SGX.

Now it gets a bit tricky -- /usr/bin/lscpu on my system is stripped off debugging symbols and even function names. Fortunately, we can search through https://github.com/util-linux/util-linux/tree/v2.38.1 repo with keywords osrelease, systab, and so on (see the Gramine log above).

This leads us to this code snippet: https://github.com/util-linux/util-linux/blob/v2.38.1/sys-utils/lscpu-virt.c#L528-L546

Looking at different functions, we end up with our suspect: is_vmware_platform(). That's what it does:

• calls vmware_bdoor(): https://github.com/util-linux/util-linux/blob/54a4d5c3ec33f2f743309ec883b9854818a25e31/sys-utils/lscpu-virt.c#L491
• which in turns has inline asm with inl instruction: https://github.com/util-linux/util-linux/blob/54a4d5c3ec33f2f743309ec883b9854818a25e31/sys-utils/lscpu-virt.c#L432

So we found the illegal instruction: it happens because lscpu tries to figure out in all ways possible whether it runs on bare metal or inside a VM. As part of this series of checks, lscpu calls is_vmware_platform() function to verify whether the VM runs using VMWare hypervisor. The verification is done through some magic I/O-port reads on port 0x5658. This is where Gramine-SGX fails.

Workaround

We cannot modify or rebuild lscpu (this would be the correct fix -- e.g., by disabling INCLUDE_VMWARE_BDOOR macro).

So we apply a workaround because we see this C snippet: https://github.com/util-linux/util-linux/blob/v2.38.1/sys-utils/lscpu-virt.c#L470-L471

In other words, if Current User ID is not zero (i.e., current user is not root), then this whole VMWare check is skipped. Thus the workaround: run Gramine with a non-zero user.

In the end, I did this change:

diff --git a/CI-Examples/bash/Makefile b/CI-Examples/bash/Makefile
@@ -17,6 +17,8 @@ bash.manifest: manifest.template
                -Darch_libdir=$(ARCH_LIBDIR) \
+               -Denv_user_uid=$(shell id -u) \
+               -Denv_user_gid=$(shell id -g) \
                $< >$@
diff --git a/CI-Examples/bash/manifest.template b/CI-Examples/bash/manifest.template
@@ -1,13 +1,16 @@
 # This is a general manifest template for running Bash and core utility programs,
 # including ls, cat, cp, date, and rm.

+loader.uid = {{ env_user_uid }}
+loader.gid = {{ env_user_gid }}
+
 loader.entrypoint = "file:{{ gramine.libos }}"

After rebuilding the Bash example, it worked:

gramine/CI-Examples/bash$ gramine-sgx bash -c scripts/lscpu_test.sh
...
Architecture:         x86_64
CPU op-mode(s):       32-bit, 64-bit
Byte Order:           Little Endian