[PAL/Linux-SGX] Introduce sgx.cpu_features.[...] manifest options

Introduce `sgx.cpu_features.[...] = "[unspecified|disabled|required]"`
instead of `sgx.require_[...] = true|false`. Also, make more explicit
the difference between security-hardening CPU features and
not-security-hardening ones.

Mapping of deprecated and new manifest options:
- `sgx.cpu_features.[...] = "unspecified"` -> `sgx.require_[...] = false`
- `sgx.cpu_features.[...] = "required"` -> `sgx.require_[...] = true`

`sgx.cpu_features.[...] = "disabled"` is new and it disables the
corresponding CPU feature inside the SGX enclave even if this CPU
feature is available on the system: this may improve enclave performance
because this CPU feature will *not* be saved and restored during enclave
entry/exit.

This commit also renames `sgx.require_exinfo` to `sgx.use_exinfo`. The
old name was ambiguous -- `require_exinfo = false` can be read as "don't
require exinfo", but in practice it disables exinfo even on CPUs which
support it.

Signed-off-by: Dmitrii Kuvaiskii <dmitrii.kuvaiskii@intel.com>

Documentation/manifest-syntax.rst

@@ -775,28 +775,75 @@ untrusted memory in potentially insecure manner - susceptible to
 CVE-2022-21233 (INTEL-SA-00657) and CVE-2022-21166 (INTEL-SA-00615)
 respectively.
 
-Optional CPU features (AVX, AVX512, MPX, PKRU, AMX, EXINFO)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+SGX EXINFO
+^^^^^^^^^^
 
 ::
 
-    sgx.require_avx    = [true|false]
-    sgx.require_avx512 = [true|false]
-    sgx.require_mpx    = [true|false]
-    sgx.require_pkru   = [true|false]
-    sgx.require_amx    = [true|false]
-    sgx.require_exinfo = [true|false]
+    sgx.use_exinfo = [true|false]
     (Default: false)
 
-This syntax ensures that the CPU features are available and enabled for the
-enclave. If the options are set in the manifest but the features are unavailable
-on the platform, enclave initialization will fail. If the options are unset,
-enclave initialization will succeed even if these features are unavailable on
-the platform.
+If ``sgx.use_exinfo`` is set, user application can retrieve faulting address in
+signal handler in case of a page fault. Otherwise (set to ``false``), the
+faulting address will always be provided as ``0``. The default is ``false``
+because some frameworks/runtimes could otherwise print the callstack and
+variables/registers on exceptions, potentially leaking data.
 
-If ``require_exinfo`` is set, user application can retrieve faulting address in
-signal handler in case of a page fault. Otherwise (set to ``false``),
-the faulting address will always be provided as ``0``.
+Optional CPU features (AVX, AVX512, AMX, MPX, PKRU)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    sgx.cpu_features.avx    = "[unspecified|disabled|required]"
+    sgx.cpu_features.avx512 = "[unspecified|disabled|required]"
+    sgx.cpu_features.amx    = "[unspecified|disabled|required]"
+    (Default: "unspecified")
+
+    sgx.cpu_features.mpx    = "[disabled|required]"
+    sgx.cpu_features.pkru   = "[disabled|required]"
+    (Default: "disabled")
+
+For the SGX threat model, Gramine divides CPU features in two subsets:
+not-security-hardening features (currently AVX, AVX512 and AMX) and
+security-hardening features (MPX and PKRU).
+
+The ``"unspecified"`` syntax applies only to not-security-hardening features. It
+means that the enclave initialization will succeed regardless of whether the CPU
+feature is available on the platform or not. The CPU features will be enabled in
+the enclave if they are available on the platform. Security-hardening features
+cannot be marked as ``"unspecified"`` because this would lead to different
+security properties of the application on different platforms.
+
+The ``"disabled"`` syntax disables the CPU feature inside the enclave even if
+this CPU feature is available on the platform. This may improve enclave
+performance because this CPU feature will *not* be saved and restored during
+enclave entry/exit. This syntax is provided to improve performance of
+applications that are known to *not* rely on certain CPU features. Be aware that
+if the application relies on some disabled CPU features, the application will
+fail with SIGILL ("illegal instruction"). For example, if the application is
+built with AVX support, and AVX is disabled in the manifest, the application
+will crash.
+
+The ``"required"`` syntax ensures that the CPU feature is available and enabled
+for the enclave. If such option is set in the manifest but the CPU feature is
+unavailable on the platform, enclave initialization will fail.
+
+When in doubt, it is recommended to keep the default values for these features.
+In this case, Gramine auto-detects the corresponding not-security-hardening CPU
+features on the platform and enables them if available, and disables
+security-hardening CPU features. This allows the SGX enclave to be executed on
+the widest range of platforms.
+
+At the technical level, the mapping from CPU features to SGX fields is as
+follows:
+
+- ``"unspecified"``: ``SIGSTRUCT.ATTRIBUTEMASK[feature] = 0``. The untrusted
+  loader of Gramine sets ``SECS.ATTRIBUTES[feature] = 0`` if it can't detect the
+  feature on the platform, and ``SECS.ATTRIBUTES[feature] = 1`` otherwise.
+- ``"required"``: ``SIGSTRUCT.ATTRIBUTEMASK[feature] = 1`` and
+  ``SIGSTRUCT.ATTRIBUTES[feature] = 1``.
+- ``"disabled"``: ``SIGSTRUCT.ATTRIBUTEMASK[feature] = 1`` and
+  ``SIGSTRUCT.ATTRIBUTES[feature] = 0``.
 
 ISV Product ID and SVN
 ^^^^^^^^^^^^^^^^^^^^^^
@@ -1134,3 +1181,47 @@ In addition, the application manifest must also contain ``sgx.debug = true``.
    independently.
 
 See :ref:`vtune-sgx-profiling` for more information.
+
+Deprecated options
+------------------
+
+Optional CPU features (deprecated syntax)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    sgx.require_avx    = [true|false]
+    sgx.require_avx512 = [true|false]
+    sgx.require_mpx    = [true|false]
+    sgx.require_pkru   = [true|false]
+    sgx.require_amx    = [true|false]
+
+This syntax specified whether to require certain CPU features to be available on
+the platform where the enclave executes. This syntax has been replaced with
+``sgx.cpu_features.[avx|avx512|mpx|pkru|amx]``.
+
+Previously, the default value for these features was ``false``. This was
+ambiguous because e.g. ``sgx.require_avx = false`` meant that the AVX feature
+was auto-detected on the platform, whereas ``sgx.require_mpx = false`` meant
+that the MPX feature was always disabled inside the enclave.
+
+.. note ::
+   Previously, security-hardening features (MPX and PKRU) were underspecified.
+   For example, ``sgx.require_mpx = false`` meant that the MPX feature was
+   always disabled inside the enclave (regardless of whether the CPU supported
+   it or not), and ``sgx.require_mpx = true`` meant that the MPX feature was
+   always enabled inside the enclave (even if the CPU did not support it, which
+   would result in enclave failures). This is in contrast to
+   non-security-hardening features (AVX, AVX512, AMX) which could be
+   "unspecified". The new manifest syntax fixes this ambiguity.
+
+SGX EXINFO (deprecated syntax)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    sgx.require_exinfo = [true|false]
+
+This syntax specified whether a user application can retrieve faulting address
+in signal handler in case of a page fault. This syntax was renamed to
+``sgx.use_exinfo``. The default value was ``false``.

Documentation/performance.rst

@@ -216,26 +216,33 @@ untrusted RPC threads on another set of cores (e.g., on second hyper-threads).
 In general, the classical performance-tuning strategies are applicable for
 Gramine and Exitless multi-threaded workloads.
 
-Optional CPU features (AVX, AVX512, MPX, PKRU, AMX)
+Optional CPU features (AVX, AVX512, AMX, MPX, PKRU)
 ---------------------------------------------------
 
 SGX technology allows to specify which CPU features are required to run the SGX
-enclave. Gramine "inherits" this and has the following manifest options:
-``sgx.require_avx``, ``sgx.require_avx512``, ``sgx.require_mpx``,
-``sgx.require_pkru``, ``sgx.require_amx``. By default, all of them are set to
-``false`` – this means that SGX hardware will allow running the SGX enclave on
-any system, whether the system has AVX/AVX512/MPX/PKRU/AMX features or not.
+enclave. Gramine "inherits" this and has the manifest options for
+AVX/AVX512/AMX/MPX/PKRU CPU features under ``sgx.cpu_features``. By default, all
+not-security-hardening CPU features (AVX, AVX512, AMX) are set to
+``"unspecified"`` -- this means that Gramine will allow running the SGX enclave
+on any platform, whether the platform has the CPU features or not.
 
 Gramine typically correctly identifies the features of the underlying platform
-and propagates the information on AVX/AVX512/MPX/PKRU/AMX inside the enclave and
-to the application. It is recommended to leave these manifest options as-is (set
-to ``false``). However, we observed on some platforms that the graminized
+and propagates the information on AVX/AVX512/AMX inside the enclave and to the
+application. It is recommended to leave these manifest options as-is (set to
+``"unspecified"``). However, we observed on some platforms that the graminized
 application cannot detect these features and falls back to a slow
 implementation. For example, some crypto libraries do not recognize AVX on the
 platform and use very slow functions, leading to 10-100x overhead over native
 (we still don't know the reason for this behavior). If you suspect this can be
-your case, enable the features in the manifest, e.g., set ``sgx.require_avx =
-true``.
+your case, enable the features in the manifest, e.g., set ``sgx.cpu_features.avx
+= "required"``.
+
+Gramine also allows to explicitly disable CPU features using the ``"disabled"``
+keyword -- this disables the corresponding CPU feature inside the SGX enclave
+even if this CPU feature is available on the system. This may improve enclave
+performance because this CPU feature will *not* be saved and restored during
+enclave entry/exit. But be aware that if the graminized application relies on
+this CPU feature, the application will crash with "illegal instruction".
 
 For more information on SGX logic regarding optional CPU features, see the Intel
 Software Developer Manual, Table 38-3 ("Layout of ATTRIBUTES Structure") under

pal/src/host/linux-sgx/sgx_arch.h

@@ -98,6 +98,11 @@ typedef uint8_t sgx_isvfamily_id_t[SGX_ISV_FAMILY_ID_SIZE];
 /* EINIT must verify most of the SECS.ATTRIBUTES[127..64] bits (XFRM/XCR0 bits) against
  * SIGSTRUCT.ATTRIBUTES[127..64].
  *
+ * This default XFRM mask may be modified via the manifest options `sgx.cpu_features.[feature]`. If
+ * the manifest option for some feature is set to "required" or "disabled", then the corresponding
+ * bits in the XFRM mask are set. If the manifest option is set to "unspecified", then the
+ * corresponding bits are unset.
+ *
  * Notes:
  *   - Verified bits include: bit 0 + bit 1 (X87 + SSE, always enabled in SGX), bit 3 + bit 4
  *     (BNDREG + BNDCSR, enables Intel MPX), bit 9 (PKRU, enables Intel MPK), and all reserved bits.

python/graminelibos/manifest.py

@@ -93,12 +93,6 @@ def __init__(self, manifest_str):
         sgx.setdefault('isvsvn', 0)
         sgx.setdefault('remote_attestation', "none")
         sgx.setdefault('debug', False)
-        sgx.setdefault('require_avx', False)
-        sgx.setdefault('require_avx512', False)
-        sgx.setdefault('require_mpx', False)
-        sgx.setdefault('require_pkru', False)
-        sgx.setdefault('require_amx', False)
-        sgx.setdefault('require_exinfo', False)
         sgx.setdefault('enable_stats', False)
         sgx.setdefault('edmm_enable', False)
 
@@ -107,6 +101,22 @@ def __init__(self, manifest_str):
         else:
             sgx.setdefault('enclave_size', DEFAULT_ENCLAVE_SIZE_NO_EDMM)
 
+        # TODO: below was deprecated in release v1.6, remove this check in v1.7
+        #       (but keep the `if` body)
+        if not 'require_exinfo' in sgx:
+            sgx.setdefault('use_exinfo', False)
+
+        # TODO: below were deprecated in release v1.6, remove this check in v1.7
+        #       (but keep the `if` body)
+        deprecated = ['require_avx', 'require_avx512', 'require_amx', 'require_mpx', 'require_pkru']
+        if not any(key in sgx for key in deprecated):
+            sgx_cpu_features = sgx.setdefault('cpu_features', {})
+            sgx_cpu_features.setdefault('avx', "unspecified")
+            sgx_cpu_features.setdefault('avx512', "unspecified")
+            sgx_cpu_features.setdefault('amx', "unspecified")
+            sgx_cpu_features.setdefault('mpx', "disabled")
+            sgx_cpu_features.setdefault('pkru', "disabled")
+
         if not isinstance(sgx['trusted_files'], list):
             raise ValueError("Unsupported trusted files syntax, more info: " +
                   "https://gramine.readthedocs.io/en/latest/manifest-syntax.html#trusted-files")

python/graminelibos/sgx_sign.py

@@ -77,36 +77,78 @@ def parse_size(value):
 def collect_bits(manifest_sgx, options_dict):
     val = 0
     for opt, bits in options_dict.items():
-        if manifest_sgx[opt] == 1:
+        if manifest_sgx.get(opt) is True:
             val |= bits
     return val
 
 
+def collect_cpu_feature_bits(manifest_cpu_features, options_dict, val, mask, security_hardening):
+    for opt, bits in options_dict.items():
+        if opt not in manifest_cpu_features:
+            continue
+        if manifest_cpu_features[opt] == "required":
+            val |= bits
+            mask |= bits
+        elif manifest_cpu_features[opt] == "disabled":
+            val &= ~bits
+            mask |= bits
+        elif security_hardening or manifest_cpu_features[opt] != "unspecified":
+            raise KeyError(f'Manifest option `sgx.cpu_features.{opt}` has a disallowed value')
+    return val, mask
+
+
 def get_enclave_attributes(manifest_sgx):
     flags_dict = {
         'debug': offs.SGX_FLAGS_DEBUG,
     }
+    flags = collect_bits(manifest_sgx, flags_dict)
+    if ARCHITECTURE == 'amd64':
+        flags |= offs.SGX_FLAGS_MODE64BIT
 
-    xfrms_dict = {
+    # TODO: 'require_exinfo' was deprecated in release v1.6, should be removed in v1.7
+    if 'require_exinfo' in manifest_sgx:
+        if 'use_exinfo' in manifest_sgx:
+            raise KeyError(f'`sgx.require_exinfo` cannot coexist with `sgx.use_exinfo`')
+        manifest_sgx['use_exinfo'] = manifest_sgx.pop('require_exinfo')
+
+    miscs_dict = {
+        'use_exinfo': offs.SGX_MISCSELECT_EXINFO,
+    }
+    miscs = collect_bits(manifest_sgx, miscs_dict)
+
+    # TODO: these were deprecated in release v1.6, so they should be removed in v1.7
+    deprecated_xfrms_dict = {
         'require_avx': offs.SGX_XFRM_AVX,
         'require_avx512': offs.SGX_XFRM_AVX512,
         'require_mpx': offs.SGX_XFRM_MPX,
         'require_pkru': offs.SGX_XFRM_PKRU,
         'require_amx': offs.SGX_XFRM_AMX,
     }
-
-    miscs_dict = {
-        'require_exinfo': offs.SGX_MISCSELECT_EXINFO,
+    xfrms_dict = {
+        'avx': offs.SGX_XFRM_AVX,
+        'avx512': offs.SGX_XFRM_AVX512,
+        'amx': offs.SGX_XFRM_AMX,
+    }
+    secure_xfrms_dict = {
+        'mpx': offs.SGX_XFRM_MPX,
+        'pkru': offs.SGX_XFRM_PKRU,
     }
 
-    flags = collect_bits(manifest_sgx, flags_dict)
-    if ARCHITECTURE == 'amd64':
-        flags |= offs.SGX_FLAGS_MODE64BIT
-
-    xfrms = offs.SGX_XFRM_LEGACY | collect_bits(manifest_sgx, xfrms_dict)
-    miscs = collect_bits(manifest_sgx, miscs_dict)
+    xfrms, xfrms_mask = offs.SGX_XFRM_LEGACY, offs.SGX_XFRM_MASK_CONST
+    if manifest_sgx.get('cpu_features') is None:
+        # collect deprecated `sgx.require_xxx` options; remove this in v1.7
+        xfrms |= collect_bits(manifest_sgx, deprecated_xfrms_dict)
+    else:
+        for deprecated_key in deprecated_xfrms_dict:
+            if deprecated_key in manifest_sgx:
+                raise KeyError(f'`sgx.cpu_features` cannot coexist with `sgx.{deprecated_key}`')
+        xfrms, xfrms_mask = collect_cpu_feature_bits(manifest_sgx['cpu_features'], xfrms_dict,
+                                                     xfrms, xfrms_mask, security_hardening=False)
+        xfrms, xfrms_mask = collect_cpu_feature_bits(manifest_sgx['cpu_features'],
+                                                     secure_xfrms_dict, xfrms, xfrms_mask,
+                                                     security_hardening=True)
 
-    return flags, xfrms, miscs
+    return flags, miscs, xfrms, xfrms_mask
 
 
 # Populate Enclave Memory
@@ -527,10 +569,11 @@ def get_tbssigstruct(manifest_path, date, libpal=SGX_LIBPAL, verbose=False):
     sig['isv_prod_id'] = manifest_sgx['isvprodid']
     sig['isv_svn'] = manifest_sgx['isvsvn']
 
-    attribute_flags, attribute_xfrms, misc_select = get_enclave_attributes(manifest_sgx)
+    attribute_flags, misc_select, attribute_xfrms, xfrms_mask = get_enclave_attributes(manifest_sgx)
     sig['attribute_flags'] = attribute_flags
-    sig['attribute_xfrms'] = attribute_xfrms
     sig['misc_select'] = misc_select
+    sig['attribute_xfrms'] = attribute_xfrms
+    sig['attribute_xfrm_mask'] = xfrms_mask
 
     return sig