Runtime detection of CPU features

[Julow]

Hi !

This PR adds runtime detection of CPU features.
Both accelerated and generic functions are built and linked. This is implemented with a global variable checked on every call.
The detection is done at initialization time using the library cpuid.

Running the all the benchmarks in bench/ either shows no difference or a slight decrease in performances (consistent 1% on sha1, too much noise on others).

I first tried using modules (on this branch) and selecting the right function at the beginning.
This was slower because calls to external functions were no longer inlined (a closure and caml_apply indirections)

The cpuid library allocates (around 3Kb) data structures that are only needed once but not collected.

I didn't remove the MIRAGE_CRYPTO_ACCELERATE env variable, the acceleration is only enabled on x86_64 systems so removing the variable would not remove any code.

Partially solves issue #45.
The RDRAND/RDSEED flags are still detected at build time but this may not be a problem for deployments as only architecture is checked, which is still in cross-compilation's scope.

[hannesm]

thanks @Julow, nice work.
some comments:

• I'd be in favour to remove the MIRAGE_CRYPTO_ACCELERATE environment variable entirely (from cfg.ml and README)
• as you mention, the aesni should only be enabled on x86 -- so could we in cfg.ml guard these flags behind the uname -m? (so they won't be enabled on ARM)
• will cpuid (which includes C code) fine on mirage / solo5?
• I'm not convinced of the extra C file misc_sse.c, at least the export_counter can be shared with misc.c
• the rdseed & rdrand CPU features are already detected at run-time, whether the compiler supported -mrdseed is discovered (by uname -m) at compile time (thus, this is fine -- it could now use cpuid in a similar fashion as you do for aes/sse/pcmulq)

[hannesm]

just two lines below, the accelerate_flags should be passed -- i.e only on x86 platform, not on arm (this is as far as I can tell what cpuid would have done as well)... or am I misguided and ARM nowadays supports -mssse3 -maes -mpclmul?

[Julow]

I was wondering the same. Fixed !
Should it be passed for amd64 and x86 ? It seems only x86_64 was selected in mirage-crypto.h. I removed that check so we have only one place to worry about this.

[Julow]

I'd be in favour to remove the MIRAGE_CRYPTO_ACCELERATE environment variable entirely (from cfg.ml and README)

Done.

as you mention, the aesni should only be enabled on x86 -- so could we in cfg.ml guard these flags behind the uname -m? (so they won't be enabled on ARM)

Currently this is guarded behind the preprocessor macro __x86_64__, which is defined by the compiler. Is the uname check necessary to build on other architectures ? (eg. what happen if we pass the -mssse3 to the compiler on ARM?)

will cpuid (which includes C code) fine on mirage / solo5?

I didn't test that yet. How can I ensures that ? Building a simple unikernel on my machine is enough ?

I'm not convinced of the extra C file misc_sse.c, at least the export_counter can be shared with misc.c

The export_counter macro is different too. I kept it to keep similarity between the two files.
I created an other file for similarity with other "accelerated" files (aes_aesni and ghash_pclmul). Should I join the files ?

the rdseed & rdrand CPU features are already detected at run-time, whether the compiler supported -mrdseed is discovered (by uname -m) at compile time (thus, this is fine -- it could now use cpuid in a similar fashion as you do for aes/sse/pcmulq)

It is also possible to do the opposite, checking all features in C, removing the cpuid library if it is problematic for portability. (or memory)

[avsm]

I think merging this may need to be dependent on @samoht's port to dune, as the cpuid code will just work then in all supported Mirage backends (since C code is compiled in a workspace automatically in @samoht's patches).

[hannesm]

Thanks @Julow

How can I ensures that ? Building a simple unikernel on my machine is enough ?

yes, that should be enough. to properly run on mirageos, C sources need to be "cross-"compiled with different headers (see the freestanding and xen subdirectories in this repository).

The export_counter macro is different too.

hmm, at a second look, it is fine to have the sources as you have in this PR.

It is also possible to do the opposite, checking all features in C

looking at entropy_cpu_stubs.c (which already reads the CPUID and evaluates the feature bits), I'm in favour of this approach: less code, less dependencies, and it doesn't look too horrible.

[Julow]

Added a commit that implements cpuid call without using the cpuid OCaml library. No longer needed to worry about cross compilation.
The code look the same and detection is still called from OCaml initialization.

[hannesm]

thanks again Julow, I pushed a single commit onto your branch, which:

• does acceleration as well if uname -m is "amd64" (as BSDs do it)
• extend and use the cpu feature detection in entropy_cpu_stubs (and guard the feature detection only by i386 || x86_64, not -DACCELERATE))

I hope these changes are fine with you, if that is the case, and CI is green, I'm eager to merge this (all squashed into a single commit). 🌴

[hannesm]

LGTM

[hannesm]

merged, the CI failure was a transient network failure, and other macOS jobs were successful.