HTTPS Get throws IOException on ARM

[schnappviech]

I'm using NET Core 2.1 on an ARM system. Normal HTTP connections do work, but a HTTPS Get will throw an IOException. Root certificates from https://curl.haxx.se/docs/caextract.html are installed to store root.

Dotnet

dotnet --info

Host (useful for support):
  Version: 2.1.6
  Commit:  3f4f8eebd8

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.1.6 [/media/maps/dotnet/shared/Microsoft.NETCore.App]

Program.cs

    using System;
    using System.Net.Http;
    
    namespace TlsTest
    {
        internal class Program
            {
            static void Main()
            {
                var client = new HttpClient();
                HttpResponseMessage resp;

                resp = client.GetAsync("http://httpbin.org/").Result;
                Console.WriteLine(resp.ToString());

                resp = client.GetAsync("https://httpbin.org/").Result;
                Console.WriteLine(resp.ToString());
            }
        }
    }

Output

    StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
    {
      Connection: keep-alive
      Server: gunicorn/19.9.0
      Date: Wed, 14 Nov 2018 16:40:46 GMT
      Access-Control-Allow-Origin: *
      Access-Control-Allow-Credentials: true
      Via: 1.1 vegur
      Content-Type: text/html; charset=utf-8
      Content-Length: 10122
    }

    Unhandled Exception: System.AggregateException: One or more errors occurred. (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The server returned an invalid or unrecognized response.
       at System.Net.Http.HttpConnection.FillAsync()
       at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
       at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
       --- End of inner exception stack trace ---
       at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
       --- End of inner exception stack trace ---
       at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at System.Threading.Tasks.Task`1.get_Result()
       at TlsTest.Program.Main()
    Aborted

Wireshark Dump of HTTPS connection

dump.https.pcap.zip

[davidsh]

@ruesselspringer

I'm using NET Core 2.1 on an ARM system

What kind of ARM system is this? Trying to determine if this is 32 or 64 bit.

[schnappviech]

I'm using the 32bit build. I should mention that the OS is build using Yocto, so it is not an officially supported distribution. The application is ported from Mono, where TLS worked well.

Are there any dependencies I can check?

[davidsh]

Wireshark Dump of HTTPS connection

The TLS connection was established successfully with the server. But then the server requested terminated of the TCP connection. See frame 13.

[schnappviech]

@davidsh Thanks for your feedback!
I have now also tried on a Raspberry Pi 2 running Raspbian Lite; The program runs flawlessly there. So the issue is certainly not related to the runtime.

By enabling logging I saw that verifying the remote certificate takes around 80 seconds. That seems to be too long for the server which closes the connection meanwhile. After adding a RemoteCertificateValidationCallback I was able to track the delay down to somewhere between lines 1011 and 1053 in System/Net/Security/SecureChannel.cs. There, a call to VerifyCertificateProperties seems to be the most likely cause for the delay; This function calls native OpenSSL functions.

I could verify this by forcing OpenSSL to use its SW implementation instead of HW acceleration. Verification is still slow then (around 35 seconds) but below the timeout of the server, so the connection succeeds.

As the current version of OpenSSL is old (1.0.1p) I will try to supply a new one. At least 1.0.2p, but do you know by chance if 1.1.1 would also be supported? I'm just wondering why the performance of Mono (using BoringSSL I believe) is so much better than OpenSSL...

Log OpenSSL HW Accelerated

11/15/2018 14:49:34 [Microsoft-System-Net-Security-2] (SafeDeleteSslContext#7167227, GetRemoteCertificate, [Version]
  V3

[Subject]
  CN=httpbin.org
  Simple Name: httpbin.org
  DNS Name: httpbin.org

<snip>

11/15/2018 14:50:03 [Microsoft-System-Net-Http-1] (HttpConnectionPoolManager#12547953, RemoveStalePools, ).
11/15/2018 14:50:03 [Microsoft-System-Net-Http-21] (9799115, 0, 0, CleanCacheAndDisposeIfUnused, HttpConnectionPoolhttp://httpbin.org:80:Cleaning pool.).
11/15/2018 14:50:03 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Poll, Interop.Winsock.select returns socketCount:0).
11/15/2018 14:50:03 [Microsoft-System-Net-Http-21] (32001227, 0, 0, CleanCacheAndDisposeIfUnused, HttpConnectionPoolhttps://httpbin.org:443:Cleaning pool.).
11/15/2018 14:50:03 [Microsoft-System-Net-Http-2] (HttpConnectionPoolManager#12547953, RemoveStalePools, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-1] (HttpConnectionPoolManager#12547953, RemoveStalePools, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-21] (9799115, 0, 0, CleanCacheAndDisposeIfUnused, HttpConnectionPoolhttp://httpbin.org:80:Cleaning pool.).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Poll, Interop.Winsock.select returns socketCount:0).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-21] (9799115, 5773521, 0, IsUsable, HttpConnection(HttpConnectionPoolhttp://httpbin.org:80): Connection no longer usable. Unexpected data received.).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-21] (9799115, 5773521, 0, Dispose, HttpConnection(HttpConnectionPoolhttp://httpbin.org:80): Connection closing.).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-21] (9799115, 0, 0, DecrementConnectionCount, HttpConnectionPoolhttp://httpbin.org:80:).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-1] (Socket#21083178, InternalShutdown, how:Both).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-1] (Socket#21083178, Close, (-1)).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Close, timeout = -1).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Dispose, timeout = -1).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-1] (Socket#21083178, Dispose, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Dispose, disposing:true CleanedUp:False).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-1] (Socket#21083178, Dispose, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-1] (Socket#21083178, SetToDisconnected, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (Socket#21083178, Dispose, Calling _handle.CloseAsIs()).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (SafeCloseSocket#55530882, CloseAsIs, _innerSocket=InnerSafeCloseSocket:13061901(0x36)).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (SafeCloseSocket#55530882, ReleaseHandle, _innerSocket=(null)).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (InnerSafeCloseSocket#13061901, ReleaseHandle, handle:0x36).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (InnerSafeCloseSocket#13061901, InnerReleaseHandle, handle:0x36 Following 'blockable' branch.).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-4] (InnerSafeCloseSocket#13061901, InnerReleaseHandle, handle:0x36, close()#1:0).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-2] (Socket#21083178, Dispose, ).
11/15/2018 14:50:33 [Microsoft-System-Net-Sockets-2] (Socket#21083178, Close, -1).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-21] (32001227, 0, 0, CleanCacheAndDisposeIfUnused, HttpConnectionPoolhttps://httpbin.org:443:Cleaning pool.).
11/15/2018 14:50:33 [Microsoft-System-Net-Http-2] (HttpConnectionPoolManager#12547953, RemoveStalePools, ).
11/15/2018 14:50:51 RemoteCertificateValidationCallback
11/15/2018 14:50:51 [Microsoft-System-Net-Security-35] (10553853).
11/15/2018 14:50:51 [Microsoft-System-Net-Security-4] (SecureChannel#10553853, VerifyRemoteCertificate, Cert validation, remote cert = [Version]

Log OpenSSL SW

11/15/2018 15:10:40 [Microsoft-System-Net-Security-2] (SafeDeleteSslContext#64254500, GetRemoteCertificate, [Version]
  V3

[Subject]
  CN=httpbin.org
  Simple Name: httpbin.org
  DNS Name: httpbin.org

<snip>

11/15/2018 15:11:06 RemoteCertificateValidationCallback
11/15/2018 15:11:06 [Microsoft-System-Net-Security-35] (26067845).
11/15/2018 15:11:06 [Microsoft-System-Net-Security-4] (SecureChannel#26067845, VerifyRemoteCertificate, Cert validation, remote cert = [Version]

[schnappviech]

Can answer by myself. OpenSSL 1.1 did not work (missing symbols), but a fresh 1.0.2p does the trick.

[davidsh]

Glad to see that you got your issues resolved!

At least 1.0.2p, but do you know by chance if 1.1.1 would also be supported?

@bartonjs - Is OpenSsl 1.1 supported on .NET Core 2.1? Or only on .NET Core 3.0?

[bartonjs]

Is OpenSsl 1.1 supported on .NET Core 2.1? Or only on .NET Core 3.0?

Only 3.0