Adapting attested_tls example

[stdevMac]

Hi guys!

This time I am trying to adapt the attested_tls example into my application, but I continue getting this error:
OE

=== _verify_identity() isvprodid = 1, isvsvn=1

 failed! tlssrv_accept returned -29312

Host: setup_tls_server failed with ecall return val: -29312
Host:  oe_enclave app exited successfully 

Client

OE enclave identity verified successfully!
Client: attestation certificate verified.
tlscli_connect failed!
INFO [11-15|15:43:55.917] server: failed to establish channel

Do you have any idea about what's happening there?

[paulcallen]

@jxyang will follow up with you to investigate.

[jxyang]

Just curious, did you sign or package your application using a newly generated private key? That will modify the MRSIGNER and cause the _verify_identity function to fail.

You can find the MRSIGNER of your signed or packaged application with:

myst dump myst/bin/<app-name>

The MRSIGNED value needs to be plugged in here:

mystikos/solutions/attested_tls/oe_enclave/enc/tlssrv.cpp

Line 683 in 9ee6352

0xca, 0x9a, 0xd7, 0x33, 0x14, 0x48, 0x98, 0x0a,

[stdevMac]

Thanks @jxyang for your quick reply, I am in the testing part, I just include the client part in the example inside my code, comment in the example the client part, and just build & run the server inside OE, get the MRSIGNER generated with make in OE, copy inside my app for build the peer_tee_identity.h file, and run it

[stdevMac]

@jxyang if you want to reply my case, you can follow the next steps:

1. Clone silkworm
2. Checkout to branch run_inside_sgx 👉 git checkout run_inside_sgx
3. Go to your copy of the Mystikos project and do cd ./solutions/attested_tls
4. Comment line 29 of the Makefile
5. make run, this will generate file app/peer_tee_identity.h that we should copy to silkworm repo in core/silkworm/secure/peer_tee_identity.h
6. make run (will only run OE part)
7. Go to silkworm cloned project,
8. mkdir empty
9. docker build ./empty/ -t cmake:ubuntu -f Dockerfile.cmake
10. myst-appbuilder -d Dockerfile -v
11. myst mkcpio appdir/ rootfs
12. myst exec-sgx --strace rootfs /src/witness 127.0.0.1

That should give the same error

[stdevMac]

Or if you thing I am missing any step let me know
Thanks in advance!

[jxyang]

@stdevMac The attest_tls example uses mutual attestation. The error message indicates the server (the executable based on OE) rejected the attestation report from the client (the app running with Mystikos), but seems the attestation of the other direction succeeded. That implies the manual steps of obtaining/copying MRSIGNER from the OE app work as expected. BTW, the sample's Makefile shows how to automate this error-prone process.

I will try to repro with your application later. Thanks for the instructions.

[vtikoo]

@stdevMac are you by any chance deploying the server behind a load balancer.

@jxyang We've seen health-check pings cause SSL handshakes to fail with a 29312(EOF) error during AKS testing.
Some more context here .

[paulcallen]

We have not forgotten about this and will get back to you when we have a change to look at this. We are currently busy trying to get a new release out so all of the devs are busy until the end of the month.

[jxyang]

Hi @stdevMac , I finally get time to run your application. It takes a long time to run appbuilder, mostly spent in building grpc. But eventually, step 10 failed with this error:

  CMake Error at CMakeLists.txt:186 (add_subdirectory):
  The source directory

    /src/mbedtls

  does not contain a CMakeLists.txt file.

Alternatively we can get more print outs from the failing OE app by replacing fprintf with printf on the lines like here. From your description, I think you did everything correctly, and there is no reason to fail. Hopefully more messages will give us more hint to debug.

[stdevMac]

Hello guys, thanks for all your replies! @vtikoo I am running this test on the exact same machine, without any connection with any other machine.

Thanks, @paulcallen For your response!

@jxyang Yeah, It takes a lot of time to build, sorry for that! the error you are getting is a little weird, if you run git submodule update --init --recursive you should find any dependence, but also, make download this dependence, I make a fresh installation of the project and the only problem I face, was solved with git sub....

I am going to try to build from scratch Mystikos and see which log we get.

Guys, I really appreciate your quick reply and all the work you are doing here, thanks for your support, I think we are close to solving an important problem thanks to all your previous work here!!

Thanks!!!

[stdevMac]

I will come in a few hours with my log or any other error I could get building Mystikos, we really need this functionality finished

[stdevMac]

Hello guys, I have tried several ways, but still at the end getting the same error sadly.

Do you have any idea about what error -29312 means in tls handshake? is the thing that are failing in the server side and I need to know what the error means

[jxyang]

I believe -29312 is the mbedtls error MBEDTLS_ERR_SSL_CONN_EOF.

https://tls.mbed.org/api/ssl_8h.html#ab82e0253b53fa62fab94b98ef9eb97f9

But the connection might be closed due to failure of hand shake, which doesn't tell us something we didn't know.

[jxyang]

@stdevMac I tried again, and still fail at step 10, albeit with a different error message:

Step 7/7 : COPY --from=base-image /src/certificate/certificate.pem /src/certificate/certificate.pem
COPY failed: stat src/certificate/certificate.pem: file does not exist

Are you able to get more information with my suggestion above?

Alternatively we can get more print outs from the failing OE app by replacing fprintf with printf on the lines like here. From your description, I think you did everything correctly, and there is no reason to fail. Hopefully more messages will give us more hint to debug.

[stdevMac]

Is weird, let me check if everything is updated

Yes, I get more details, but running with --strace:

logs.txt

The important I think is:

=== SYS_write(fd=2110125533 buf=0x12f43aea8 count=45): pid=101 tid=101
    SYS_write(): return=-EBADF(-9): pid=101 tid=101
=== SYS_fcntl(fd=2110125533 cmd=3(F_GETFL) arg=011): pid=101 tid=101
    SYS_fcntl(): return=-EBADF(-9): pid=101 tid=101
=== SYS_shutdown(sockfd=0 how=2): pid=101 tid=101
    SYS_shutdown(): return=-ENOTSOCK(-88): pid=101 tid=101
=== SYS_close(fd=0): pid=101 tid=101
    SYS_close(): return=0(0): pid=101 tid=101
=== SYS_writev(fd=1 iov=0x13235c6a0 iovcnt=2): pid=101 tid=101
mbedtls_ssl_handshaketlscli_connect failed!

[stdevMac]

@jxyang Sorry, I update the repo, you should not have any problem now

[stdevMac]

I mean building the app

[jxyang]

Thanks @stdevMac . I am able to build silkworm rootfs now.

My debugging shows the client prematurely closed the socket before the server gets a chance to verify the client credentials. That explains the error code -29312 (mbedtls error MBEDTLS_ERR_SSL_CONN_EOF) on the server side.

On the client side, mbedtls didn't print the error until I run it inside myst-gdb. Here are the relevant error messages:

INFO [12-07|23:12:30.651] Processing witness
Running inside SGX!
Now, for secrets!
INFO [12-07|23:12:30.806] The block process fine
Starting connectionInitialize the cli structureStarting mbedtls_*Client: Received TLS certificate.
  crt->version = 3 cert_size = 5084

=== _verify_identity()
Unique ID = 2a62d82578b16652ef7933eb844fcc353a46764658c99604bac97422b6813dfa
Signer ID = e68af26aa42fce554f36e07751e2813454d5929661212bfe3c148167de22d9de


OE enclave identity verified successfully!
Client: attestation certificate verified.
_mbedtls_dbg.cli: /src/mbedtls/library/ssl_tls.c:3574: mbedtls_ssl_flush_output() returned -69 (-0x0045)
_mbedtls_dbg.cli: /src/mbedtls/library/ssl_tls.c:3417: ssl_write_record() returned -69 (-0x0045)
_mbedtls_dbg.cli: /src/mbedtls/library/ssl_tls.c:6611: mbedtls_ssl_write_handshake_msg() returned -69 (-0x0045)
mbedtls_ssl_handshaketlscli_connect failed!
server: failed to establish channel

In case you are curious, error code -69 is MBEDTLS_ERR_NET_SEND_FAILED. This confirms that the underlying net socket has been closed while the client was verifying the server credentials.

[jxyang]

The solution attested_tls uses Alpine Linux as the base image, and a musl-based mbedtls library, as shown in https://github.com/deislabs/mystikos/blob/main/solutions/attested_tls/app-dockerfile.

You may want to try the solution with Ubuntu/glibc to confirm it still works. If it does, you could copy the code under attested_tls/app into your application as it was to isolate the causes of the failure.

[jxyang]

FYI I have converted attested_tls to be a Ubuntu app, and made some simplifications. It works.

https://github.com/jxyang/mystikos/blob/attested_tls_ubuntu/solutions/attested_tls/Makefile

[stdevMac]

mmm. Thanks @jxyang, cool the changes for running inside Ubuntu, I didn't notice that part, totally my bad.

Maybe can be a difference between the library we are using for mbedtls inside C++ code....I am going to make some changes to that, what do you think? can this be the cause?

[jxyang]

Yes, I noticed that the mbedtls version included in your project is earlier than the one included in Mystikos.

I think it would be a good idea to link your application with libmbedtls-dev rather than building it from source.

[stdevMac]

@jxyang Effectively it was the error, I didn't reply earlier trying to send some info from one side to the other on the enclave.

I am getting the next error:

*** kernel panic: enter.c(853): myst_enter_kernel(): myst_exec() failed, ret=-2
0x1005ef7b2: __myst_panic()
0x1005bc8b7: myst_enter_kernel()
Makefile:17: recipe for target 'run' failed
make: *** [run] Error 139

The only change that I do is try to call the method tlscli_write.

Do you have any idea about the error I am getting?

Thank you!!! The first one is solved, we were able to make attestation! Let's go for the next goal!

[jxyang]

@stdevMac Unfortunately this is a very generic error message. There could be multiple reasons behind it.

Could you generate a STRACE and ETRACE for the execution? The command would be like:

myst exec-sgx --strace --etrace ...

[stdevMac]

Hey I manage to make it work, it was an error from my side building the message, thank you!

[jxyang]

I am glad it works for you now. @stdevMac