Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mbedtls problem with libcurl port to ESP32 #1327

Closed
nkolban opened this issue Mar 12, 2017 · 8 comments
Closed

mbedtls problem with libcurl port to ESP32 #1327

nkolban opened this issue Mar 12, 2017 · 8 comments
Labels

Comments

@nkolban
Copy link

nkolban commented Mar 12, 2017

I am porting libcurl to the ESP32 environment. For non SSL operations, no issues at all. While testing SSL with mbedtls (which is what is supplied with the ESP32 development kit), I found that a handshake fails with a network level error of -76 which appears to be a timeout.

My logic is as follows:

curlHandle = curl_easy_init();
::curl_easy_setopt(curlHandle, CURLOPT_URL, "https://httpbin.org/get");
::curl_easy_setopt(curlHandle, CURLOPT_VERBOSE, true);
::curl_easy_setopt(curlHandle, CURLOPT_ERRORBUFFER, errbuf);
::curl_easy_setopt(curlHandle, CURLOPT_SSL_VERIFYPEER, false);
::curl_easy_setopt(curlHandle, CURLOPT_CAINFO, nullptr);
int rc = ::curl_easy_perform(curlHandle);

The error reported by libcurl is:

ssl_handshake returned - mbedTLS: (-0x004C) UNKNOWN ERROR CODE (004C)

I then switched on tracing at the mbedtls level and captured the trace posted at the end of this report.

This problem is fully reproducible on my ESP32 environment. If needed, I can live chat and/or screen share as needed. The version of libcurl is the Github master as of 2017-03-12.

D (7153) curl_main_test: Testing curl ...
* timeout on name lookup is not supported
*   Trying 104.31.86.204...
* TCP_NODELAY set
* Connected to requestb.in (104.31.86.204) port 443 (#0)
* mbedTLS: Connecting to requestb.in:443
* mbedTLS: Set min SSL version to TLS 1.0
* => handshake
* client state: 0
* => flush output
* <= flush output
* client state: 1
* => flush output
* <= flush output
* => write client hello
* client hello, max version: [3:3]
* client hello, current time: 6
* dumping 'client hello, random bytes' (32 bytes)
* 0000:  00 00 00 06 88 1c 73 eb 72 28 6b 8f d0 96 7c 27  ......s.r(k...|'
* 0010:  f7 20 df 37 d1 24 d0 9d f2 07 1c ba 13 e4 3a 7b  . .7.$........:{
* client hello, session id len.: 0
* dumping 'client hello, session id' (0 bytes)
* client hello, add ciphersuite: c02c
* client hello, add ciphersuite: c030
* client hello, add ciphersuite: 009f
* client hello, add ciphersuite: c0ad
* client hello, add ciphersuite: c09f
* client hello, add ciphersuite: c024
* client hello, add ciphersuite: c028
* client hello, add ciphersuite: 006b
* client hello, add ciphersuite: c00a
* client hello, add ciphersuite: c014
* client hello, add ciphersuite: 0039
* client hello, add ciphersuite: c0af
* client hello, add ciphersuite: c0a3
* client hello, add ciphersuite: c087
* client hello, add ciphersuite: c08b
* client hello, add ciphersuite: c07d
* client hello, add ciphersuite: c073
* client hello, add ciphersuite: c077
* client hello, add ciphersuite: 00c4
* client hello, add ciphersuite: 0088
* client hello, add ciphersuite: c02b
* client hello, add ciphersuite: c02f
* client hello, add ciphersuite: 009e
* client hello, add ciphersuite: c0ac
* client hello, add ciphersuite: c09e
* client hello, add ciphersuite: c023
* client hello, add ciphersuite: c027
* client hello, add ciphersuite: 0067
* client hello, add ciphersuite: c009
* client hello, add ciphersuite: c013
* client hello, add ciphersuite: 0033
* client hello, add ciphersuite: c0ae
* client hello, add ciphersuite: c0a2
* client hello, add ciphersuite: c086
* client hello, add ciphersuite: c08a
* client hello, add ciphersuite: c07c
* client hello, add ciphersuite: c072
* client hello, add ciphersuite: c076
* client hello, add ciphersuite: 00be
* client hello, add ciphersuite: 0045
* client hello, add ciphersuite: c008
* client hello, add ciphersuite: c012
* client hello, add ciphersuite: 0016
* client hello, add ciphersuite: 00ab
* client hello, add ciphersuite: c0a7
* client hello, add ciphersuite: c038
* client hello, add ciphersuite: 00b3
* client hello, add ciphersuite: c036
* client hello, add ciphersuite: 0091
* client hello, add ciphersuite: c091
* client hello, add ciphersuite: c09b
* client hello, add ciphersuite: c097
* client hello, add ciphersuite: c0ab
* client hello, add ciphersuite: 00aa
* client hello, add ciphersuite: c0a6
* client hello, add ciphersuite: c037
* client hello, add ciphersuite: 00b2
* client hello, add ciphersuite: c035
* client hello, add ciphersuite: 0090
* client hello, add ciphersuite: c090
* client hello, add ciphersuite: c096
* client hello, add ciphersuite: c09a
* client hello, add ciphersuite: c0aa
* client hello, add ciphersuite: c034
* client hello, add ciphersuite: 008f
* client hello, add ciphersuite: 009d
* client hello, add ciphersuite: c09d
* client hello, add ciphersuite: 003d
* client hello, add ciphersuite: 0035
* client hello, add ciphersuite: c032
* client hello, add ciphersuite: c02a
* client hello, add ciphersuite: c00f
* client hello, add ciphersuite: c02e
* client hello, add ciphersuite: c026
* client hello, add ciphersuite: c005
* client hello, add ciphersuite: c0a1
* client hello, add ciphersuite: c07b
* client hello, add ciphersuite: 00c0
* client hello, add ciphersuite: 0084
* client hello, add ciphersuite: c08d
* client hello, add ciphersuite: c079
* client hello, add ciphersuite: c089
* client hello, add ciphersuite: c075
* client hello, add ciphersuite: 009c
* client hello, add ciphersuite: c09c
* client hello, add ciphersuite: 003c
* client hello, add ciphersuite: 002f
* client hello, add ciphersuite: c031
* client hello, add ciphersuite: c029
* client hello, add ciphersuite: c00e
* client hello, add ciphersuite: c02d
* client hello, add ciphersuite: c025
* client hello, add ciphersuite: c004
* client hello, add ciphersuite: c0a0
* client hello, add ciphersuite: c07a
* client hello, add ciphersuite: 00ba
* client hello, add ciphersuite: 0041
* client hello, add ciphersuite: c08c
* client hello, add ciphersuite: c078
* client hello, add ciphersuite: c088
* client hello, add ciphersuite: c074
* client hello, add ciphersuite: 000a
* client hello, add ciphersuite: c00d
* client hello, add ciphersuite: c003
* client hello, add ciphersuite: 00ad
* client hello, add ciphersuite: 00b7
* client hello, add ciphersuite: 0095
* client hello, add ciphersuite: c093
* client hello, add ciphersuite: c099
* client hello, add ciphersuite: 00ac
* client hello, add ciphersuite: 00b6
* client hello, add ciphersuite: 0094
* client hello, add ciphersuite: c092
* client hello, add ciphersuite: c098
* client hello, add ciphersuite: 0093
* client hello, add ciphersuite: 00a9
* client hello, add ciphersuite: c0a5
* client hello, add ciphersuite: 00af
* client hello, add ciphersuite: 008d
* client hello, add ciphersuite: c08f
* client hello, add ciphersuite: c095
* client hello, add ciphersuite: c0a9
* client hello, add ciphersuite: 00a8
* client hello, add ciphersuite: c0a4
* client hello, add ciphersuite: 00ae
* client hello, add ciphersuite: 008c
* client hello, add ciphersuite: c08e
* client hello, add ciphersuite: c094
* client hello, add ciphersuite: c0a8
* client hello, add ciphersuite: 008b
* client hello, got 131 ciphersuites
* client hello, compress len.: 1
* client hello, compress alg.: 0
* client hello, adding server name extension: requestb.in
* client hello, adding signature_algorithms extension
* client hello, adding supported_elliptic_curves extension
* client hello, adding supported_point_formats extension
* client hello, adding encrypt_then_mac extension
* client hello, adding extended_master_secret extension
* client hello, total extension length: 88
* => write record
* output record: msgtype = 22, version = [3:1], msglen = 395
* dumping 'output record sent to network' (400 bytes)
* 0000:  16 03 01 01 8b 01 00 01 87 03 03 00 00 00 06 88  ................
* 0010:  1c 73 eb 72 28 6b 8f d0 96 7c 27 f7 20 df 37 d1  .s.r(k...|'. .7.
* 0020:  24 d0 9d f2 07 1c ba 13 e4 3a 7b 00 01 06 c0 2c  $........:{....,
* 0030:  c0 30 00 9f c0 ad c0 9f c0 24 c0 28 00 6b c0 0a  .0.......$.(.k..
* 0040:  c0 14 00 39 c0 af c0 a3 c0 87 c0 8b c0 7d c0 73  ...9.........}.s
* 0050:  c0 77 00 c4 00 88 c0 2b c0 2f 00 9e c0 ac c0 9e  .w.....+./......
* 0060:  c0 23 c0 27 00 67 c0 09 c0 13 00 33 c0 ae c0 a2  .#.'.g.....3....
* 0070:  c0 86 c0 8a c0 7c c0 72 c0 76 00 be 00 45 c0 08  .....|.r.v...E..
* 0080:  c0 12 00 16 00 ab c0 a7 c0 38 00 b3 c0 36 00 91  .........8...6..
* 0090:  c0 91 c0 9b c0 97 c0 ab 00 aa c0 a6 c0 37 00 b2  .............7..
* 00a0:  c0 35 00 90 c0 90 c0 96 c0 9a c0 aa c0 34 00 8f  .5...........4..
* 00b0:  00 9d c0 9d 00 3d 00 35 c0 32 c0 2a c0 0f c0 2e  .....=.5.2.*....
* 00c0:  c0 26 c0 05 c0 a1 c0 7b 00 c0 00 84 c0 8d c0 79  .&.....{.......y
* 00d0:  c0 89 c0 75 00 9c c0 9c 00 3c 00 2f c0 31 c0 29  ...u.....<./.1.)
* 00e0:  c0 0e c0 2d c0 25 c0 04 c0 a0 c0 7a 00 ba 00 41  ...-.%.....z...A
* 00f0:  c0 8c c0 78 c0 88 c0 74 00 0a c0 0d c0 03 00 ad  ...x...t........
* 0100:  00 b7 00 95 c0 93 c0 99 00 ac 00 b6 00 94 c0 92  ................
* 0110:  c0 98 00 93 00 a9 c0 a5 00 af 00 8d c0 8f c0 95  ................
* 0120:  c0 a9 00 a8 c0 a4 00 ae 00 8c c0 8e c0 94 c0 a8  ................
* 0130:  00 8b 00 ff 01 00 00 58 00 00 00 10 00 0e 00 00  .......X........
* 0140:  0b 72 65 71 75 65 73 74 62 2e 69 6e 00 0d 00 16  .requestb.in....
* 0150:  00 14 06 03 06 01 05 03 05 01 04 03 04 01 03 03  ................
* 0160:  03 01 02 03 02 01 00 0a 00 18 00 16 00 19 00 1c  ................
* 0170:  00 18 00 1b 00 17 00 16 00 1a 00 15 00 14 00 13  ................
* 0180:  00 12 00 0b 00 02 01 00 00 16 00 00 00 17 00 00  ................
* => flush output
* message length: 400, out_left: 400
* ssl->f_send() returned 400 (-0xfffffe70)
* <= flush output
* <= write record
* <= write client hello
* client state: 2
* => flush output
* <= flush output
* => parse server hello
* => read record
* => fetch input
* in_left: 0, nb_want: 5
* in_left: 0, nb_want: 5
* ssl->f_recv(_timeout)() returned -76 (-0x004c)
* mbedtls_ssl_fetch_input() returned -76 (-0x004c)
* mbedtls_ssl_read_record() returned -76 (-0x004c)
* <= handshake
* ssl_handshake returned - mbedTLS: (-0x004C) UNKNOWN ERROR CODE (004C)
* Closing connection 0
E (8089) RESTClient: get(): ssl_handshake returned - mbedTLS: (-0x004C) UNKNOWN ERROR CODE (004C)
@jay jay added the TLS label Mar 12, 2017
@jay
Copy link
Member

jay commented Mar 12, 2017

  • ssl_handshake returned - mbedTLS: (-0x004C) UNKNOWN ERROR CODE (004C)

That error is MBEDTLS_ERR_NET_RECV_FAILED. The "UNKNOWN ERROR CODE" is coming from mbedtls_strerror, it is what is returned when the error string for an error is not compiled in. In this case mbedTLS must be built with MBEDTLS_NET_C for the expected error string to be returned to libcurl. If your mbedTLS was not built with that then networking will not work without custom callbacks, apparently.

So I think it's very unlikely libcurl is at fault here. Can you try the same thing with the mbedTLS ssl_client1 sample program, with a few adjustments:

diff --git a/programs/ssl/ssl_client1.c b/programs/ssl/ssl_client1.c
index fa70431..8bcf70a 100644
--- a/programs/ssl/ssl_client1.c
+++ b/programs/ssl/ssl_client1.c
@@ -62,9 +62,9 @@ int main( void )
 
 #include <string.h>
 
-#define SERVER_PORT "4433"
-#define SERVER_NAME "localhost"
-#define GET_REQUEST "GET / HTTP/1.0\r\n\r\n"
+#define SERVER_PORT "443"
+#define SERVER_NAME "requestb.in"
+#define GET_REQUEST "GET / HTTP/1.1\r\nHost: requestb.in\r\nConnection: Close\r\n\r\n"
 
 #define DEBUG_LEVEL 1
 

That should return the page and then say EOF, error MBEDTLS_ERR_SSL_CONN_EOF (-0x7280), and press a key to exit.

Also re libcurl you should pass the options types exactly as specified, in other words for options that require a type long value don't pass true or false, instead pass 1L or 0L.

@jay
Copy link
Member

jay commented Mar 12, 2017

I should have mentioned libcurl does set the custom callbacks for send and recv, see https://github.com/curl/curl/blob/curl-7_53_1/lib/vtls/mbedtls.c#L368-L371. So it turns out the error message just isn't built in but that likely (may? I'm really not sure) has nothing to do with why you are seeing that error.

@nkolban
Copy link
Author

nkolban commented Mar 13, 2017

Howdy ... I validated that the distribution of mbedtls being used has MBEDTLS_NET_C defined. I also changed the options types to 1L and 0L as appropriate.

Searching the internet, I found the following:

https://tls.mbed.org/discussions/bug-report-issues/ssl_write-and-ssl_read-timeout

This could be a clue. Apparently we can get an error like this is we call mbed to receive data and there is no data to read. Since libcurl is the caller of mbed, that might point to libcurl as a potential issue.

Running ESP32 supplied mbedtls samples seems to show that mbedtls as a framework works. If there is any other tests or diagnostics I can execute, please don't hesitate to let me know and I'll try and run them as quickly as possible.

19:31(Local) 2017-03-12
Digging down ... found that error code -0x004C comes from this line:

https://github.com/ARMmbed/mbedtls/blob/master/include/mbedtls/net_sockets.h#L42

which is defined as the symbol MBEDTLS_ERR_NET_RECV_FAILED. Digging on that, we find this reference:

https://github.com/ARMmbed/mbedtls/blob/master/library/net_sockets.c#L485

19:49(Local) 2017-3-12
I added some printf() statements before the returns of this error code to log the underlying errno and found that it is 11 which is EAGAIN from the read() system call on the socket.
By digging deeper ... I believe this to be a code returned from a read of a socket that if flagged as non-blocking when it would otherwise block. Does libcurl set its sockets to be non-blocking? My belief is that the socket needed by mbedtls needs to be a blocking socket.

Searching even further ... I found this release note for libcurl ...

https://curl.haxx.se/mail/archive-2017-02/0020.html

Which contains a line that reads ...

mbedTLS: fix multi interface non-blocking handshake [30]

Might this issue be another instance of that?

see also: #1223

@jay
Copy link
Member

jay commented Mar 13, 2017

Please make new posts instead of amending the last one like that, you asked a few more questions that nobody receiving e-mail updates would see since github only sends out the unedited original.

Does libcurl set its sockets to be non-blocking?

Usually yes, though there are some cases where the socket is temporarily set blocking. In the case of mbedtls handshake it is non-blocking.

I added some printf() statements before the returns of this error code to log the underlying errno and found that it is 11 which is EAGAIN from the read() system call on the socket.

read returning -1 and errno EAGAIN in mbedTLS is normal. What is not normal though is why that would result in mbedTLS's recv wrapper returning MBEDTLS_ERR_NET_RECV_FAILED. You referred to this section in mbedtls_net_recv (win32 stuff removed):

    if( ret < 0 )
    {
        if( net_would_block( ctx ) != 0 )
            return( MBEDTLS_ERR_SSL_WANT_READ );

        if( errno == EPIPE || errno == ECONNRESET )
            return( MBEDTLS_ERR_NET_CONN_RESET );

        if( errno == EINTR )
            return( MBEDTLS_ERR_SSL_WANT_READ );

        return( MBEDTLS_ERR_NET_RECV_FAILED );
    }

So net_would_block returns 0 which causes MBEDTLS_ERR_NET_RECV_FAILED. net_would_block is (win32 stuff removed):

static int net_would_block( const mbedtls_net_context *ctx )
{
    /*
     * Never return 'WOULD BLOCK' on a non-blocking socket
     */
    if( ( fcntl( ctx->fd, F_GETFL ) & O_NONBLOCK ) != O_NONBLOCK )
        return( 0 );

    switch( errno )
    {
#if defined EAGAIN
        case EAGAIN:
#endif
#if defined EWOULDBLOCK && EWOULDBLOCK != EAGAIN
        case EWOULDBLOCK:
#endif
            return( 1 );
    }
    return( 0 );
}

So maybe EAGAIN isn't defined for that unit or for some reason the socket wasn't set non-blocking by libcurl? check that return, sprinkle in some printfs see what happens

mbedTLS: fix multi interface non-blocking handshake [30]

Might this issue be another instance of that?

Tthere was a bug that was fixed in 7.53 so that it would operate properly in non-blocking mode. You are using master and if that had something to do with it I think we'd see a different code path.

Running ESP32 supplied mbedtls samples seems to show that mbedtls as a framework works.

Please run the sample I linked to if you can. Also do other https websites show the same issue, and can you give us your curl_version() please. Thanks

@jay
Copy link
Member

jay commented Mar 13, 2017

Also check curlx_nonblock and see how exactly it's setting nonblock, maybe the way mbedtls is checking it is different and not compatible for some reason.

@jay
Copy link
Member

jay commented Mar 13, 2017

I realize now it doesn't make any sense that a blocking socket would return EAGAIN... so I'd look into the other things first

@nkolban
Copy link
Author

nkolban commented Mar 13, 2017

@jay,
MANY thanks for your guidance and ultra quick response. I have spent the evening digging and have found the cause. The problem does not live in libcurl nor does it live in mbedtls ... and nor does it live in my own logic.

Let me take the time to explain. My target for the port of libcurl is the ESP32 platform. This is a WiFi enabled MCU device (google ESP32). It supports the majority of the POSIX APIs and a TCP/IP stack based on LWIP. There is a development kit for the ESP32 called ESP-IDF. This kit supplies the source of the libraries that comprise the vast majority of the run time platform on which applications are written.

The core of the problem reported in this issue, and exactly as you diagnosed, is the function net_would_block. If we look at the current source version of this function from mbedtls we find the following:

https://github.com/ARMmbed/mbedtls/blob/master/library/net_sockets.c#L270

All good.

Now if we look at the ESP-IDF version, we find a subtle difference:

https://github.com/espressif/esp-idf/blob/master/components/mbedtls/port/net.c#L204

The difference is that the ESP-IDF version doesn't look at "errno" from the environment but instead "asks the socket" for its last error. The reason being that the ESP32 is a multi-threaded preemptive environment and the use of the "global" errno shouldn't be used (which makes sense). However, look again at the ESP-IDF logic. The pertinent part I summarize below:

if ( ( fcntl( ctx->fd, F_GETFL, 0) & O_NONBLOCK ) != O_NONBLOCK ) {
   return ( 0 );
}

int error = mbedtls_net_errno(ctx->fd);

The intent here in the if statement is to check if the socket is NOT non blocking ... and since the socket IS non-blocking we fall through to the call to mbedtls_net_errno() which checks the last error reported on the socket. And HERE is where things went south. The call to fcntl (i.e. the if statement reset the error ... and hence the result is that we lost the errno we were looking for).

I will now follow up with a report against ESP-IDF. MANY thanks for the assistance.

Neil (a friend)

@nkolban nkolban closed this as completed Mar 13, 2017
@jay
Copy link
Member

jay commented Mar 13, 2017

No problem, thanks for the update.

Ref: espressif/esp-idf#424

@lock lock bot locked as resolved and limited conversation to collaborators May 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Development

No branches or pull requests

2 participants