C# Android SSL Crash using System.Net.Http.HttpClient

[jpetersen23]

Godot version

4.2-beta4

System information

Godot v4.2.beta4.mono - macOS 13.6.0 - Vulkan (Mobile) - integrated Apple M2 Max - Apple M2 Max (12 Threads)

Issue description

Making a network request using System.Net.Http.HttpClient to "https://www.example.com" cause a crash, calling "http://www.example.com" does not crash. If I use Godot's node HttpRequest it does not crash.

The logcat output for the crash looks like:

11-06 17:44:57.813 3989 4048 I godot : Pressed button SystemSsl
11-06 17:44:57.856 3989 4048 I godot : send request to https://www.example.com
11-06 17:44:57.902 4405 4405 I SKBD : and isTosAccept false
11-06 17:44:58.071 1061 1209 D DnsProxyListener: DNSDBG::dns addrinfo af 0
11-06 17:44:58.103 4405 4405 I SKBD : and isTosAccept false
11-06 17:44:58.282 3989 4201 E godot : USER ERROR: BUG: Unreferenced static string to 0: Physics2DConstraintSolveIslands
11-06 17:44:58.282 3989 4201 E godot : at: unref (core/string/string_name.cpp:127)
11-06 17:44:58.282 3989 4201 E godot : USER ERROR: BUG: Unreferenced static string to 0: Physics2DConstraintSetup
11-06 17:44:58.282 3989 4201 E godot : at: unref (core/string/string_name.cpp:127)
--------- beginning of crash
11-06 17:44:58.284 3989 4201 F libc : FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x6f51b6400c)

Steps to reproduce

This is the code that crashes in its simplest form:

private void Send(string url)
{
	var client = new System.Net.Http.HttpClient();
	var request = new HttpRequestMessage(HttpMethod.Get, url);
	GD.Print($"send request to {url}");
	var response = client.Send(request); // <-- crashes on this line.
	GD.Print(response.ToString()); // this does not get reached.
	var stream = response.Content.ReadAsStream();
	StreamReader reader = new StreamReader(stream, System.Text.Encoding.UTF8);
	var text = reader.ReadToEnd();
	GD.Print(text);
}

public override void _Process(double delta)
{
	Send("https://www.example.com");
}

Minimal reproduction project

The sample project has 4 buttons, each doing a different network call to example.com:
SystemSSL
SystemNonSSL
GodotSSL
GodotNonSSL

The system ones use System.Net.Http.HttpClient, the Godot ones use a HttpRequest node.

SSLCrashExample_Buttons_For_Zipping.zip

[jpetersen23]

Of course as soon as I post this, I find dotnet/runtime#94230 which may or may not be related given that Android should be using dotnet7

[simonrozsival]

@jpetersen23 I don't think it's the same issue, I'm able to reproduce it on Android API 29 emulator. Unfortunately, there isn't much information in the logs so it's hard to pinpoint where the bug originates.

[ozanyasindogan]

I'm having the same issue while trying to use Grpc.Net.Client to connect to a TLS backed Grpc server. The game crashes on a physical Android device with Android v11 (API level 30). I have a .Net MAUI application which uses the same library and works fine on Android devices. I have below trace log while debugging with Android Studio:

2023-11-21 02:56:37.964 14063-14063 com.example.grpctest com.example.grpctest W type=1400 audit(0.0:665913): avc: granted { execute } for comm=2E4E455420546872656164506F6F6C path="/data/data/com.example.grpctest/cache/data_GrpcTest_android_arm64/libSystem.Security.Cryptography.Native.OpenSsl.so" dev="sdc46" ino=524684 scontext=u:r:untrusted_app:s0:c139,c257,c512,c768 tcontext=u:object_r:app_data_file:s0:c139,c257,c512,c768 tclass=file 2023-11-21 02:56:37.968 14063-14157 godot com.example.grpctest E USER ERROR: BUG: Unreferenced static string to 0: Physics2DConstraintSolveIslands 2023-11-21 02:56:37.968 14063-14157 godot com.example.grpctest E at: unref (core/string/string_name.cpp:127) 2023-11-21 02:56:37.968 14063-14157 godot com.example.grpctest E USER ERROR: BUG: Unreferenced static string to 0: Physics2DConstraintSetup 2023-11-21 02:56:37.968 14063-14157 godot com.example.grpctest E at: unref (core/string/string_name.cpp:127) 2023-11-21 02:56:37.973 14063-14157 libc com.example.grpctest A FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x756d6fb294)

So the warning here:
type=1400 audit(0.0:665913): avc: granted { execute } for comm=2E4E455420546872656164506F6F6C path="/data/data/com.example.grpctest/cache/data_GrpcTest_android_arm64/libSystem.Security.Cryptography.Native.OpenSsl.so" dev="sdc46" ino=524684 scontext=u:r:untrusted_app:s0:c139,c257,c512,c768 tcontext=u:object_r:app_data_file:s0:c139,c257,c512,c768 tclass=file

could give a hint for the reason of the crash.
The game works fine on Windows by the way, Grpc client successfully connects and send request,
receive response in the expected way.

What do you think this could be related to?

[sanform]

Same issue but coming from the C# PlayFabSDK. I can't seem to get any debug info or call stack. I used debug lines to find that the failure occurred at the PostAsync() call.

While groping in the dark I tried this suggestion: https://stackoverflow.com/questions/48475378/httpclient-not-working-on-android

But still no dice.

[ozanyasindogan]

Same issue but coming from the C# PlayFabSDK. I can't seem to get any debug info or call stack. I used debug lines to find that the failure occurred at the PostAsync() call.

While groping in the dark I tried this suggestion: https://stackoverflow.com/questions/48475378/httpclient-not-working-on-android

But still no dice.

Any type of SSL stream made by .Net fails actually. I tried using raw sockets with SSLStream and the app crashed again.
Only Godot's inbuilt socket functions are working properly, .Net sockets with SSL/TLS authentication fails. I built my own game server in .Net as a worker service but I think I will rewrite my client code to use Godot's internal networking functions now. What I don't like there in the examples, altough I might be wrong, I'm just discovering this area in Godot (networking), is that Godot examples seems to be polling the stream for new data in every frame rather than creating a new thread with a blocking read call and it seems to work in the main thread. I might be wrong, I'm still getting into it but I wouldn't like to manage networking in the main thread for sure. I wish they could have fixed that SSLStream issue, it would be even possible to use gRPC with .Net and it would ease my job a lot rather than writing my own own protocol with headers, contents, frames, chunks etc.. I tested gRPC, it actually works well with Godot Windows apps, even works on Android but if I don't use a secure stream. So the problem surely lies on the SSL side of Godot while publishing to Android. Did anyone tested it on iOS/MacOS?

[akoeplinger]

This looks like you're missing OpenSSL (libssl.so) which the libSystem.Security.Cryptography.Native.OpenSsl.so library depends on.

[ozanyasindogan]

This looks like you're missing OpenSSL (libssl.so) which the libSystem.Security.Cryptography.Native.OpenSsl.so library depends on.

Well actually when I unzip the APK, I see that libSystem.Security.Cryptography.Native.OpenSsl.so is actually packaged under assets/.godot/mono/publish/arm64. I've attached the directory listing. But like you said, I don't see libssl.so here. So if it is depending on it, that makes perfect sense that the app crashes.

dir.txt

[ozanyasindogan]

This looks like you're missing OpenSSL (libssl.so) which the libSystem.Security.Cryptography.Native.OpenSsl.so library depends on.

Well actually when I unzip the APK, I see that libSystem.Security.Cryptography.Native.OpenSsl.so is actually packaged under assets/.godot/mono/publish/arm64. I've attached the directory listing. But like you said, I don't see libssl.so here. So if it is depending on it, that makes perfect sense that the app crashes.

dir.txt

This looks like you're missing OpenSSL (libssl.so) which the libSystem.Security.Cryptography.Native.OpenSsl.so library depends on.

I just checked the APK of my MAUI project which uses Grpc and couldn't locate libssl.so there as well.
There is a document from google stating that developers should reference public API's instead of private ones and
Android will not be allowing that in future releases >24+. I don't know if it is the reason why it is not packaged
in both MAUI project and Godot, but I'm sure my MAUI project works perfectly fine with Android, even on HTTP3.

[akoeplinger]

@ozanyasindogan that's expected, MAUI doesn't use the linux-bionic runtime identifier but android which relies on a Java class to handle crypto operations instead of OpenSSL.

The doc you linked talks about not referencing the libssl which ships as part of Android, but you're fine to bring your own (with all the caveats that entails).

[akien-mga]

@ozanyasindogan Please have patience. Android support for C# in 4.2 is experimental, and some issues like this are expected and will be fixed in due time, once contributors have time to debug the issue and provide a fix.

[ozanyasindogan]

@ozanyasindogan Please have patience. Android support for C# in 4.2 is experimental, and some issues like this are expected and will be fixed in due time, once contributors have time to debug the issue and provide a fix.

Thank you very much Rémi @akien-mga for given attention, I really appreciate all the work done by Godot team to create such a great product. Also thank you very much Alexander @akoeplinger for your valuable comments. It really encourages me as a .Net developer that MS staff is supporting the use of .Net in various platforms.

[akoeplinger]

One thing you can try is downloading and extracting https://dl.google.com/android/maven2/com/android/ndk/thirdparty/openssl/1.1.1p-beta-1/openssl-1.1.1p-beta-1.aar and packaging libssl.so and libcrypto.so into the app and see if that helps.

[spirifoxy]

One thing you can try is downloading and extracting https://dl.google.com/android/maven2/com/android/ndk/thirdparty/openssl/1.1.1p-beta-1/openssl-1.1.1p-beta-1.aar and packaging libssl.so and libcrypto.so into the app and see if that helps.

Can confirm that it helps to fix the problem.

[TCROC]

@akoeplinger @spirifoxy how do I go about packaging that with the android build? Do I package it as an Android plugin? https://docs.godotengine.org/en/stable/tutorials/platform/android/android_plugin.html

Or do I somehow configure .net to use that .so?

[TCROC]

I found something interesting I'll try tomorrow: dotnet/maui#6174 (comment)

[TCROC]

For some reason <UseNativeHttpHandler> doesn't seem to be taking, but I really would've expected it to work based on microsoft's docs:

https://github.com/dotnet/runtime/blob/9b57a265c7efd3732b035bade005561a04767128/docs/workflow/trimming/feature-switches.md

My csproj currently looks like:

<Project Sdk="Godot.NET.Sdk/4.3.0-dev">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <!-- Android Start -->
    <TargetFramework Condition=" '$(GodotTargetPlatform)' == 'android' ">net7.0</TargetFramework>
    <!-- <PublishTrimmed Condition=" '$(GodotTargetPlatform)' == 'android' ">true</PublishTrimmed>
    <TrimMode Condition=" '$(GodotTargetPlatform)' == 'android' ">full</TrimMode> -->
    <UseNativeHttpHandler Condition=" '$(GodotTargetPlatform)' == 'android' ">false</UseNativeHttpHandler>
    <!-- Android End -->
    <TargetFramework Condition=" '$(GodotTargetPlatform)' == 'ios' ">net8.0</TargetFramework>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <DefaultItemExcludes>$(DefaultItemExcludes);submodules\**;other\**</DefaultItemExcludes>
    <!-- Disable warning about classes not being in a namespace -->
    <NoWarn>CA1050</NoWarn>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.GameLift" Version="3.7.201.26" />
    <PackageReference Include="MQTTnet" Version="4.3.1.873" />
    <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageReference Include="PlayFabAllSDK" Version="1.163.230915" />
    <PackageReference Include="MessagePack" Version="2.5.124" />
  </ItemGroup>
</Project>

Settings UseNativeHttpHandler doesn't seem to make a difference true or false. It did work for this person here: dotnet/maui#6174 (comment)

[TCROC]

One thing you can try is downloading and extracting https://dl.google.com/android/maven2/com/android/ndk/thirdparty/openssl/1.1.1p-beta-1/openssl-1.1.1p-beta-1.aar and packaging libssl.so and libcrypto.so into the app and see if that helps.

Manually packaging as suggested is making progress. Now I get:"The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot"

But the good news is its no longer crashing. Now to figure out how to properly include a certificate chain that makes it happy. This is an area I'm not familiar with. And for anyone wondering how to manually include those libraries:

Download and extract the openssl lib from here: https://dl.google.com/android/maven2/com/android/ndk/thirdparty/openssl/1.1.1p-beta-1/openssl-1.1.1p-beta-1.aar

Then once you extract it, you should have a folder structure similar to:

Then drag those into your AndroidManifest directory:

Then when you export, they should be included in the apk. Now to figure out the cert issues! :)

[TCROC]

Made a new finding which I think is good news. The certificate error originates on the .NET side. Was able to confirm by setting .NET to accept all certs:

		HttpClientHandler handler = new HttpClientHandler();
		handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) =>
		{
			// Implement your custom certificate validation logic here
			return true; // Return true to accept the certificate
		};

		this._client = new HttpClient(handler);

Specifically, I tested it with PlayFab as @sanform pointed out like this:

public class MyPlayFabPollyHttp : PlayFabPollyHttp
{
	public MyPlayFabPollyHttp() : base() 
	{
		HttpClientHandler handler = new HttpClientHandler();
		handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) =>
		{
			// Implement your custom certificate validation logic here
			return true; // Return true to accept the certificate
		};

		this._client = new HttpClient(handler);
	}
}

And then created an instance of the playfabClient like so:

			PluginManager.SetPlugin(new MyPlayFabPollyHttp(), PluginContract.PlayFab_Transport);
			playFabClient = new PlayFabClientInstanceAPI(new PlayFabApiSettings
			{
				TitleId = PlayFabUtils.TITLE_ID,
			});

Then my mqtt client crashed with cert errors (probably for the same reason). So gonna go see if simply accepting all certs gets me to the next step there as well.

THIS IS NOT A SOLUTION

This opens the application up to man in the middle attacks. Don't use this as the final solution. I'm merely pointing out that the next issue appears to be on the .NET side. Probably as simple as reading in and making use of the certs godot has in the repo.

[TCROC]

This looks like you're missing OpenSSL (libssl.so) which the libSystem.Security.Cryptography.Native.OpenSsl.so library depends on.

How did you determine this was what was causing the crash? The application gets farther now but then crashes. Would like to determine what is causing the current crash :)

[TCROC]

I've determined that the mqtt client seems to be working properly when accepting all certs. It may be something else causing my game to crash on Android at this point. I'll post more updates here if they are related and if not, I'll open a new issue. Assuming there is an issue on the Godot end of course.

I'll also post an update with how I officially handle the certs when I get around to it if there isn't an official solution on here by then.

[ozanyasindogan]

I've determined that the mqtt client seems to be working properly when accepting all certs. It may be something else causing my game to crash on Android at this point. I'll post more updates here if they are related and if not, I'll open a new issue. Assuming there is an issue on the Godot end of course.

I'll also post an update with how I officially handle the certs when I get around to it if there isn't an official solution on here by then.

I hope this issue will be fixed by this: #88803
From the thread I see that there is a pull request already. Hopefully that would solve this
annoying and very essential problem for C# users.

[TCROC]

Perfect! I'll merge that into my fork tomorrow and see how it goes. Thanks @ozanyasindogan! :) It's a perfect birthday present! 😊

[ozanyasindogan]

Perfect! I'll merge that into my fork tomorrow and see how it goes. Thanks @ozanyasindogan! :) It's a perfect birthday present! 😊

Happy birthday :)

[akoeplinger]

FWIW UseNativeHttpHandler only works when targetting a real android RID, not linux-bionic like godot is doing. The OpenSSL-based handler is the only one available there. MAUI is using a different setup.

[TCROC]

Gotcha. That makes sense. Thank you for that info! :)