Ensure enclave code can be reproducibly built

[tiziano88]

We should have a way of testing that building the enclave code (trusted application) on the same machine at different times, and also on different machines, results in the same (bitwise) artifact.

[tiziano88]

This is the result of building the same artifact twice on the same machine, and diffing the output using diffoscope: https://filebin.net/aefkq1sewn9zcu80/enclave-diff.html?t=etgmxyhs

@anghelcovici @annasapek @daviddrysdale

[tiziano88]

Possibly relevant: google/asylo@9c557c2 cc @deeglaze

[deeglaze]

Linkstamping is indeed a cause for different bits for identity build inputs, which is why we disabled it for reliable trust-building on attestable bits. You will have to use the same compiler and compiler flags to get the same bits, so please consistently use the distributed Asylo toolchain. If you have different toolchains straddling internal (e.g., corp build system) and external (github.com/google/asylo/tree/master/asylo/distrib/toolchain), then you're going to have a tough time getting the same bits out.

[tiziano88]

Thanks @deeglaze ! We are indeed using the Asylo toolchain, in fact running everything in the Asylo Docker image, but getting different binaries even when rebuilding twice in a row on the same machine and with the same Docker image.

[tiziano88]

FWIW I am now specifying the stamp = 0 parameter to the sgx_enclave Bazel rule (since AFAICT the new version of Asylo in which it is the default has not been released yet), but still seeing discrepancies building on the same machine.

[deeglaze]

That is not good indeed! I will take this as an issue with Asylo and see what we can do about it. Do you Considering how Bazel caches its build artifacts, I'm not confident that we'd have a simple way to create a regression test for this. build, sha256, clean, build, sha256, compare, perhaps? A few experiments I just ran with some of our test enclaves haven't discovered discrepancies, so it might just be specific to y'all's code using -frandom-seed or __FILE__ or some such bitwise reproducibility problems. If you have a repro you can share, please do share it with us on https://github.com/google/asylo/issues

[tiziano88]

Indeed that's exactly what I'm doing in tiziano88@b621c57 to check for this behaviour. Is that enough information for you to reproduce the issue? Is there anything I can try to narrow down the issue? e.g. should I specify stamp = 0 on all the rules the enclave code depends on, or is it sufficient to just do that at the top level?

[deeglaze]

stamp = 0 is for linking the final binary together, so there shouldn't be a need add linkstamping to dependency libraries. A way I'd probably go about narrowing down what could be a problematic dependency to try to build different libraries into a do-nothing enclave and see which libraries could be introducing the non-determinism.

[tiziano88]

Thanks, I'll try that, though that worries me as I guess it means that any dependency may reintroduce non-determinism at any point in the future?

[deeglaze]

If your dependencies don't have bitwise reproducible builds due to their methods of compilation, then you'll be hard-pressed to get reproducible builds when you depend on them. For now, instead of comparing checksums of the entire ELF files, may I suggest that you focus your tests on the measured bits of the ELF file as exposed in the sgx_generate_sigstruct rule? You'll get full machine paths in debug information, so you're best off comparing your -c opt builds' sigstruct material for cross-machine reproducibility.

[deeglaze]

Whatever you find is the problem, please let us know and we'll start compiling a list of debug steps for Asylo users to try when building enclaves for attestable bits.

[tiziano88]

Status update: so far I tried removing dependencies on external source (mainly wabt), but the resulting binaries are still not reproducible, so it must be something internal in the repository, or maybe some of the direct dependencies (protobuf, asylo). Will keep looking.

[tiziano88]

I have now stripped it down to essentially an empty EnclaveApplication, and still not getting reproducibility :/

[tiziano88]

I am still comparing binary hashes though. @deeglaze is this not expected to be reproducible at all? Will try with sigstruct now anyways.

[tiziano88]

@deeglaze to use the sgx_generate_sigstruct I had to bump the version of asylo we depend on to google/asylo@5ab253e (unreleased), but now it seems asylo does not compile at all:

ERROR: /.cache/bazel/_bazel_tzn/b6bc0828d5b43c7085b650e0698bf388/external/com_google_asylo/asylo/util/BUILD:270:1: C++ compilation of rule '@com_google_asylo//asylo/util:status' failed (Exit 1) x86_64-elf-g++ failed: error executing command external/com_google_asylo_toolchain/toolchain/bin/x86_64-elf-g++ -isystemasyl
o/platform/posix/include -isystemasylo/platform/system/include ... (remaining 50 argument(s) skipped)                                                      
                                                                                                                                                
Use --sandbox_debug to see verbose messages from the sandbox                                                         
In file included from external/com_google_asylo/asylo/util/error_space.h:25:0,                                                                                                                                                                                                                                               
                 from external/com_google_asylo/asylo/util/status_error_space.h:22,                                                                         
                 from external/com_google_asylo/asylo/util/status_error_space.cc:19:                             
external/com_google_asylo/asylo/platform/common/static_map.h:167:36: error: expected ';' at end of member declaration                                       
     explicit ValueInserter(T *value) ABSL_LOCKS_EXCLUDED(StaticMap::mu_) {                                                                                                      
                                    ^                                                                                                                       
external/com_google_asylo/asylo/platform/common/static_map.h:167:69: error: 'asylo::StaticMap::mu_' is not a type                                               
     explicit ValueInserter(T *value) ABSL_LOCKS_EXCLUDED(StaticMap::mu_) {                                                                                      
                                                                     ^~~                                                                                                         
external/com_google_asylo/asylo/platform/common/static_map.h:167:72: error: ISO C++ forbids declaration of 'ABSL_LOCKS_EXCLUDED' with no type [-fpermissive]                  
     explicit ValueInserter(T *value) ABSL_LOCKS_EXCLUDED(StaticMap::mu_) {                                                                                                                                                                                                                                                  

[tiziano88]

The issue with the new version of Asylo was solved by bumping version of absl too, and it seems the root cause of the reproducibility issue is google/asylo#41. I will try again once that is fixed and update this issue.

[tiziano88]

#861