sgx-sdk, sgx-psw: improve samples

[veehaitch]

Motivation for this change

Make it easier to review updates to sgx-{sdk,psw} on machines with
actual SGX hardware support. The passthru tests build and run the SGX
samples in simulation mode which works without any hardware support. To
run the samples on a machine with SGX hardware support, issue the
following command:

 $(nix-build -A sgx-sdk.runTestsHW)/bin/run-tests-hw

Make sure the SGX AESM daemon is running as some tests require it. See
the services.aesmd.* NixOS module options and the sgx-psw package
for details.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[Mic92]

Unrelated to this PR with the new systemd version from here, aesmd fails with:

Jan 10 02:18:33 turingmachine systemd[75093]: aesmd.service: Failed to set up mount namespacing: File exists
Jan 10 02:18:33 turingmachine systemd[75093]: aesmd.service: Failed at step NAMESPACE spawning /nix/store/n6mxmq4cc5apw9az1cz6qmidf7w4nj93-copy-aesmd-data-files.sh: File exists

[Mic92]

I am using the mainline sgx kernel driver
but my system somehow does not create those character devices.
So I add the following:

  systemd.tmpfiles.rules = [
    "c /dev/sgx_enclave 777 root root - 10:56"
  ];

This allows aesmd to start:

Jan 10 02:19:16 turingmachine systemd[1]: Starting Intel Architectural Enclave Service Manager...
Jan 10 02:19:16 turingmachine systemd[1]: Started Intel Architectural Enclave Service Manager.
Jan 10 02:19:16 turingmachine aesm_service[2555]: The path of system bundle: System Bundle
Jan 10 02:19:16 turingmachine aesm_service[2555]: ecdsa_quote_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: epid_quote_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: le_launch_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: linux_network_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: pce_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: quote_ex_service_bundle_name:2.0.0
Jan 10 02:19:16 turingmachine aesm_service[2555]: system_bundle:4.0.0

However the hardware test don't quite work:

sudo strace -o /tmp/strace.log -f $(nix-build -A sgx-sdk.runTestsHW)/bin/run-tests-hw
*** Running test Cxx11SGXDemo-HW ***

[get_driver_type /build/source/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.
Info: Please make sure SGX module is enabled in the BIOS, and install SGX driver afterwards.
Error: Invalid SGX device.
Enter a character before exit ...

Sgx is enabled in BIOS on my thinkpad x13 (Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz)

$ uname -a
Linux turingmachine 5.15.11-zen1 #1-NixOS ZEN SMP Tue Jan 1 00:00:00 UTC 1980 x86_64 GNU/Linux
$ dmesg | grep sgx
[    0.508793] sgx: EPC section 0xa0200000-0xa5f7ffff

Here is the strace output:

https://dl.thalheim.io/c4uivVoPbawekUjR-0MXhg/strace.txt

[Mic92]

Anything else needed?

[veehaitch]

Thanks for taking a look!

I am using the mainline sgx kernel driver
but my system somehow does not create those character devices.

I cannot quite understand what's happening. I suppose something's not working properly with regard to the setup of the SGX devices on your machine; both /dev/sgx_enclave and /dev/sgx_provision should exist if you're using the in-tree Linux SGX kernel driver.

Here is the strace output:

https://dl.thalheim.io/c4uivVoPbawekUjR-0MXhg/strace.txt

Apparently, although the in-tree Kernel driver uses the flat paths /dev/sgx_*, the SGX SDK/PSW seems to look for the enclave device within /dev/sgx/* only. According to your strace, the SDK doesn't try opening /dev/sgx_enclave:

25617 openat(AT_FDCWD, "/dev/sgx/enclave", O_RDWR) = -1 ENOENT (No such file or directory)
25617 openat(AT_FDCWD, "/dev/isgx", O_RDWR) = -1 ENOENT (No such file or directory)
25617 openat(AT_FDCWD, "/dev/sgx", O_RDWR) = -1 ENOENT (No such file or directory)

I can confirm this on my machine and also in the source code:

https://github.com/intel/linux-sgx/blob/2ee53db4e8fd25437a817612d3bcb94b66a28373/psw/urts/linux/edmm_utility.cpp#L102-L108

This is an issue as systemd decided to not provide a symlink (which seems like a sane choice): systemd/systemd#18944 (comment). As we do not install the udev rules packaged with sgx-psw (see #148593 (comment)), we also do not create any symlink even if the AESM service is enabled; creating an SGX enclave should be possible without AESM anyways. On Debian/Ubuntu, Intel advises to depend on libsgx-enclave-common which would install the same udev rules.

I'm going to open an upstream issue if you share my analysis. I'm not sure how to tackle this issue in the meantime. A tmpfiles rule would do the job:

{
  systemd.tmpfiles.rules = [
    "L /dev/sgx/enclave - - - - /dev/sgx_enclave"
  ];
}

Edit: opened an upstream issue: intel/linux-sgx#772

[Mic92]

Intel also suggests some udev rules to create those symlinks: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/94e327a8235bc7c85353dc5e0a6dc1564e6c96c6/driver/linux/README.kernel.md#udev-rules

I switched from enabled in bios to software-controlled and then enabled it with https://github.com/intel/sgx-software-enable
this gave me sgx_provision and sgx_vepc but not sgx_enclave:

$ ls -la /sys/class/misc | grep sgx
lrwxrwxrwx 0 root 10 Jan 17:08 sgx_provision -> ../../devices/virtual/misc/sgx_provision/
lrwxrwxrwx 0 root 10 Jan 17:08 sgx_vepc -> ../../devices/virtual/misc/sgx_vepc/

pretty weird.

[veehaitch]

Intel also suggests some udev rules to create those symlinks: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/94e327a8235bc7c85353dc5e0a6dc1564e6c96c6/driver/linux/README.kernel.md#udev-rules

That's correct; yet, they also mention that installing those udev rules mainly aims at supporting existing enclaves compiled with an SGX PSW without in-tree support:

To remap those nodes to be compatible with existing user space software (e.g., Intel SGX PSW 2.12 or earlier), as root user, add a new udev rules file and activate it as following.

I'm happy to open a PR which includes a udev rule to symlink /dev/sgx_{enclave,provison}.

[veehaitch]

Edit: opened an upstream issue: intel/linux-sgx#772

Intel announced that they'd address the issue in the upcoming release.

Meanwhile, I've opened #154550 which adds two udev rules to symlink the application and provisioning enclave devices for backward compatibility.

I guess this PR already showed its usefulness. Unfortunately, @Mic92's SGX setup seems not working properly (for whatever reason). Maybe anybody else with an SGX-enabled machine (preferably running NixOS) could test this PR to get it merged? @sbellem @citadelcore @arcz

[sbellem]

@veehaitch I'll try to test this out before the end of this week.

[Mic92]

At some point I might be able to get access got some other SGX hardware.

[Mic92]

Btw.: https://www.bleepingcomputer.com/news/security/new-intel-chips-wont-play-blu-ray-disks-due-to-sgx-deprecation/

[veehaitch]

Btw.: https://www.bleepingcomputer.com/news/security/new-intel-chips-wont-play-blu-ray-disks-due-to-sgx-deprecation/

Also relevant: intel/linux-sgx#760 (comment)

[veehaitch]

Did you have chance to test this, @sbellem?

[jtraue]

I ran this on my i5-8500 using 5.15.12. Looks fine until it comes to the remote attestation test:

Info: ProtobufDemoEnclave successfully returned.

*** Running test RemoteAttestation-HW ***


First round, we will try ECDSA algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Info, call sgx_select_att_key_id fail, current platform configuration doesn't support this attestation key ID. [main]
Second round, we will try EPID algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Call sgx_select_att_key_id success.
Call sgx_create_enclave success.
Call enclave_init_ra success.
Error, call sgx_ra_get_msg1_ex fail [main].
Call enclave_ra_close success.
Enter a character before exit ...

When I remove this test, all the others succeed.
@veehaitch Neither dmesg nor aesmd contain any helpful information. Can you guide me in debugging this?

[veehaitch]

@veehaitch Neither dmesg nor aesmd contain any helpful information. Can you guide me in debugging this?

Unfortunately, I don't know. Maybe it's a network issue?

[Mic92]

Maybe tcpdump helps?

[veehaitch]

I think we can merge this. I would assume the issue @jtraue sees is related to their configuration. In any case, this PR is an improvement.

[sbellem]

@veehaitch I'm trying to test it right now.

Can the hardware-based tests work with out-of-tree drivers? That is, instead of using /dev/sgx_* it would use /dev/isgx.

Nevermind, the above, it works with /dev/isgx (OOT driver) as well.

Hardware tests pass on my end.

Looks all good to me! Thanks @veehaitch!