Crash in dotnet 5.0 on Ubuntu Groovy

[qpernil]

Hi,
I have been using Pkcs11Interop to test a pkcs#11 library, and it has been working great. Recently I updated from dotnet core 3.1 to dotnet 5.0 (as they call it now, it's still dotnet core) as Ubuntu Groovy had silently updated my dotnet installation (via snap). Once I did that I seem to be getting segmentation faults before even calling any functions in the pkcs11 library (I get no debugging output from it). In fact I just tested using the wrong library name and it still segfaults. Running the same on macos and dotnet 5.0 still works.

[jariq]

Can you please try to load any other PKCS#11 library e.g. PKCS11-MOCK? There might be a conflict between the native dependencies of .NET 5 runtime and unmanaged library you are loading.

[jariq]

In fact I just tested using the wrong library name and it still segfaults.

I did not notice this comment the first time I was reading your post 🤔

I see two options now:

1. Attach C# debugger and step through Pkcs11Interop code to see which line causes the issue.
2. Generate core dump and analyze it - https://jimrich.sk/basic-analysis-of-segmentation-fault-on-linux-platforms/

[qpernil]

I have tried #2 above and it tells me very little. There seems to be no debugging info in the dotnet-generated binary. What I did get is
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000007fffffff in ?? ()

What I did notice is also that it crashes even if I just do a Console.WriteLine on the first line of Main and then return...
In fact I don't even get any printout from Console.WriteLine.

The program I'm testing with is here https://github.com/qpernil/pkcs11-tester

BR
Per

[qpernil]

I have now tested on Ubuntu Xenial, Bionic, Focal and Groovy. It fails the same way except in one instance, Bionic, where I can successfully run the application if I call dotnet run, but not if I do dotnet /path-to-dll/pkcs11-tester.dll, in which case if crashes the same as on other platforms.. Very weird. Also, I want to point out that I have a different dotnet application I can run successfully on all those machines. That one is also using p-invoke. Weird..
I am running dotnet 5.0.100 from /snap/dotnet-sdk/105/sdk/5.0.100/

[jariq]

I just installed Ubuntu 20.10 and "classic" packaged version of .NET 5.0 SDK from https://packages.microsoft.com/ubuntu/20.10/prod groovy main repository and there's no segfault on my system (see output below) so I'm guessing that the issue you're experiencing might be specific only to SNAP (AFAIK sandboxed) version of .NET 5.0.

Anyway there seems to be a problem with libdl resolution which can be fixed in quick'n'dirty way with a symlink (see output below) but I'll rather take a look at #160 and that shiny new DllImportResolver as it might offer a more elegant solution.

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dpkg -l | grep dotnet
ii  dotnet-apphost-pack-5.0                    5.0.0-1                             amd64        Microsoft.NETCore.App.Host 5.0.0
ii  dotnet-host                                5.0.0-1                             amd64        Microsoft .NET Host - 5.0.0
ii  dotnet-hostfxr-5.0                         5.0.0-1                             amd64        Microsoft .NET Host FX Resolver - 5.0.0 5.0.0
ii  dotnet-runtime-5.0                         5.0.0-1                             amd64        Microsoft .NET Runtime - 5.0.0 Microsoft.NETCore.App 5.0.0
ii  dotnet-runtime-deps-5.0                    5.0.0-1                             amd64        dotnet-runtime-deps-5.0 5.0.0
ii  dotnet-sdk-5.0                             5.0.100-1                           amd64        Microsoft .NET SDK 5.0.100
ii  dotnet-targeting-pack-5.0                  5.0.0-1                             amd64        Microsoft.NETCore.App.Ref 5.0.0

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ which dotnet
/usr/bin/dotnet

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet --version
5.0.100

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet build
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /home/jariq/dev/pkcs11-tester-master/pkcs11-tester.csproj (in 118 ms).
  pkcs11-tester -> /home/jariq/dev/pkcs11-tester-master/bin/Debug/net5.0/pkcs11-tester.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.30

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet run
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'libdl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibdl: cannot open shared object file: No such file or directory
   at Net.Pkcs11Interop.Common.NativeMethods.dlopen(String filename, Int32 flag)
   at Net.Pkcs11Interop.Common.UnmanagedLibrary.Load(String fileName)
   at Net.Pkcs11Interop.LowLevelAPI80.Pkcs11Library..ctor(String libraryPath, Boolean useGetFunctionList)
   at Net.Pkcs11Interop.HighLevelAPI80.Pkcs11Library..ctor(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI80.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Pkcs11Tester.Program.Main(String[] args) in /home/jariq/dev/pkcs11-tester-master/Program.cs:line 32

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ sudo ln /usr/lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so
[sudo] password for jariq:

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet run
Unhandled exception. Net.Pkcs11Interop.Common.UnmanagedException: Unable to load library
   at Net.Pkcs11Interop.Common.UnmanagedLibrary.Load(String fileName)
   at Net.Pkcs11Interop.LowLevelAPI80.Pkcs11Library..ctor(String libraryPath, Boolean useGetFunctionList)
   at Net.Pkcs11Interop.HighLevelAPI80.Pkcs11Library..ctor(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI80.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Pkcs11Tester.Program.Main(String[] args) in /home/jariq/dev/pkcs11-tester-master/Program.cs:line 32

[jariq]

Segfaults on my system too with .NET 5 installed from SNAP but only after I created that quick'n'dirty libdl symlink:

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ snap list | grep dotnet
dotnet-sdk         5.0.100             105   latest/stable    dotnetcore*  classic

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ which dotnet
/snap/bin/dotnet

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet --version
5.0.100

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet build
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /home/jariq/dev/pkcs11-tester-master/pkcs11-tester.csproj (in 125 ms).
  pkcs11-tester -> /home/jariq/dev/pkcs11-tester-master/bin/Debug/net5.0/pkcs11-tester.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.34

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet run
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'libdl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibdl: cannot open shared object file: No such file or directory
   at Net.Pkcs11Interop.Common.NativeMethods.dlopen(String filename, Int32 flag)
   at Net.Pkcs11Interop.Common.UnmanagedLibrary.Load(String fileName)
   at Net.Pkcs11Interop.LowLevelAPI80.Pkcs11Library..ctor(String libraryPath, Boolean useGetFunctionList)
   at Net.Pkcs11Interop.HighLevelAPI80.Pkcs11Library..ctor(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI80.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Net.Pkcs11Interop.HighLevelAPI.Factories.Pkcs11LibraryFactory.LoadPkcs11Library(Pkcs11InteropFactories factories, String libraryPath, AppType appType, InitType initType)
   at Pkcs11Tester.Program.Main(String[] args) in /home/jariq/dev/pkcs11-tester-master/Program.cs:line 32

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ sudo ln /usr/lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ dotnet run

jariq@ubuntu2010:~/dev/pkcs11-tester-master$ tail -f /var/log/syslog 
Nov 18 22:05:29 ubuntu2010 dbus-daemon[702]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Nov 18 22:05:29 ubuntu2010 systemd[1]: Started PackageKit Daemon.
Nov 18 22:05:37 ubuntu2010 systemd[1447]: tracker-extract.service: Succeeded.
Nov 18 22:05:59 ubuntu2010 tracker-store[30215]: OK
Nov 18 22:05:59 ubuntu2010 systemd[1447]: tracker-store.service: Succeeded.
Nov 18 22:05:59 ubuntu2010 systemd[1]: Starting Message of the Day...
Nov 18 22:05:59 ubuntu2010 systemd[1]: motd-news.service: Succeeded.
Nov 18 22:05:59 ubuntu2010 systemd[1]: Finished Message of the Day.
Nov 18 22:08:37 ubuntu2010 kernel: [ 1480.933943] pkcs11-tester[31219]: segfault at 0 ip 00007fa64004540c sp 00007ffc1d046f50 error 4 in libdl-2.32.so[7fa640045000+2000]
Nov 18 22:08:37 ubuntu2010 kernel: [ 1480.933948] Code: 44 24 10 48 8b 4c 24 28 64 48 2b 0c 25 28 00 00 00 75 30 48 83 c4 38 c3 66 0f 1f 44 00 00 48 8b 05 b9 3c 00 00 48 8b 54 24 38 <48> 8b 00 48 8b 4c 24 28 64 48 2b 0c 25 28 00 00 00 75 06 48 83 c4

[jariq]

I managed to resolve my segfault with .NET 5 installed from SNAP, quick'n'dirty libdl symlink NOT created and following code added to your pkcs11-tester tool:

using System.Reflection;
using System.Runtime.InteropServices;

static IntPtr CustomDllImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? dllImportSearchPath)
{
	string mappedLibraryName = (libraryName == "libdl") ? "libdl.so.2" : libraryName;
	return NativeLibrary.Load(mappedLibraryName, assembly, dllImportSearchPath);
}

static void Main(string[] args)
{
	NativeLibrary.SetDllImportResolver(typeof(Pkcs11InteropFactories).Assembly, CustomDllImportResolver);
	
	...
}

@qpernil can you please try whether this fixes your issue?

[qpernil]

If I use dotnet 5.0 from the (apparently broken earlier but corrected today) microsoft 20.10 package everything works fine like before. So it would seem the snap is currently broken.

Your custom mapping thing doesn't seem to be working as it seems the NativeLibrary.Load appends another .so to the end of the library name, and thus it fails. I could see this by setting the env var LD_DEBUG=libs. But it's not needed with the working dotnet version.

Thanks for your support on this !

[qpernil]

Right now there is no link to the 20.10 packages file, I just took the url for Focal and changed 20.04 to 20.10 and that worked, I got Groovy apt packages. When I tried with the Focal stuff it failed to load the library, probably the same isssue you were seeing above even though I didn't see the message about libdl not being found.

[qpernil]

So I'm happy to close this issue now unless you want to use it to investigate more. Thx again ! Pkcs11Interop is very useful.

[qpernil]

Tested that it works as before on xenial, bionic, focal and groovy.

[jariq]

OK. I'm closing this issue as I'm unable to reproduce your segfault 😉

[qpernil]

A final note - I discovered that if you use the NativeLibrary.Load with one argument it uses your library name as-is, and if you use the one with three args it does platform-specific 'decorating' so that foo becomes libfoo.so etc