PortableThreadPool can't initialize in DPDK app

[landerverhacklansweeper]

Description

The PortableThreadPool doesn't work when also using Data Plane Development Kit (DPDK).

Any attempt to run C# code that requires the ThreadPool crashes with the following trace:

Unhandled Exception: System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Threading.Thread.StartCore() + 0xa5
   at System.Threading.PortableThreadPool.GateThread.CreateGateThread() + 0x92

This is part of a C# library built as a Navtive .so lib for Linux.

Reproduction Steps

Create a simple C# library that uses a Task.

public static class ThreadPoolIssue
{
    [UnmanagedCallersOnly(EntryPoint = "test_task")]
    public static void TestTask()
    {
        Console.WriteLine("Starting task");
        Task.Delay(1000).ContinueWith(t =>
        {
            Console.WriteLine("Task completed after 1 second delay");
        });
    }
}

Publish the library using AOT for Linux
lin-x64.pubxml

<?xml version="1.0" encoding="utf-8"?>
<Project>
	<PropertyGroup>
		<Configuration>Release</Configuration>
		<Platform>Any CPU</Platform>
		<PublishDir>..\publish\</PublishDir>
		<PublishProtocol>FileSystem</PublishProtocol>
		<_TargetId>Folder</_TargetId>
		<TargetFramework>net9.0</TargetFramework>
		<RuntimeIdentifier>linux-x64</RuntimeIdentifier>
		<SelfContained>true</SelfContained>
		<PublishAot>true</PublishAot>
	</PropertyGroup>
</Project>

dotnet publish -p:PublishProfile=lin-x64.pubxml

Create sample C application
testapp.h

#ifndef TESTAPP_H
#define TESTAPP_H

#include <stdint.h>
#include <stdbool.h>

#ifdef __cplusplus
extern "C" {
#endif

void test_task();

#ifdef __cplusplus
}
#endif

#endif // TESTAPP_H

testapp.c

#include "testapp.h"
#include <unistd.h> // Include this header for sleep

int main(int argc, char* argv[])
{
	test_task();
	sleep(10); // Sleep for 10 seconds to allow threads to complete

	return 0;
}

Compile the C application (adjust paths if needed)

gcc -g -o ./publish/testapp ./ThreadPoolIssueLib/testapp.c -L./publish -lThreadPoolIssue -ldl -lpthread -lm -Wl,-rpath='$ORIGIN'

Verify that the application run outside of DPDK (this should work):

cd ./publish
./testapp

set up DPDK:
https://learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu

Run code inside DPDK application --> see crash
(https://core.dpdk.org/doc/quick-start/)
Even though the creation of the GateThread fails, I can see the output of the Task after one second.

Expected behavior

The thread pool should initialize correctly. The application should not crash.

Actual behavior

Crashes with the following StackTrace:

Unhandled Exception: System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Threading.Thread.StartCore() + 0xa5
   at System.Threading.PortableThreadPool.GateThread.CreateGateThread() + 0x92

The task does seem to run though The stdout does show "Task completed after 1 second delay" after one second.

Regression?

No response

Known Workarounds

No response

Configuration

I tested multiple combinations. All with the same behavior.

Ubuntu 18.04 and 24.04 (x64)
.NET SDK 8.0.411, 9.0.301
.NET runtimes Microsoft.NETCore.App 8.0.17, Microsoft.NETCore.App 9.0.6
DPDK 21.11.0

Other information

• I believe the OutOfMemoryException is incorrect. The system has plenty of memory left. It is the GateThread that fails to start and I believe the runtime simply assumes that it's because of out of memory. See Failing to create a thread shouldn't assume that this an out of memory issue #71761

• Creating regular threads just works, I've made 32 with success.

    [UnmanagedCallersOnly(EntryPoint = "test_thread")]
    public static void TestTread()
    {
        Console.WriteLine("Starting thread");
        Thread thread = new(() =>
        {
            Console.WriteLine("Thread started");
            for (int i = 0; i < 5; i++)
            {
                Console.WriteLine($"Thread running iteration {i + 1}");
                Thread.Sleep(1000);
            }
            Console.WriteLine("Thread completed");
        });
        thread.Start();
    }

• DPDK uses core pinning
• DPDK uses huge pages
• Any code that doesn't use the thread pool seems to run fine

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

Is it possible to capture an strace of the execution under DPDK? It would probably point to the failing syscall.

[landerverhacklansweeper]

mmap(NULL, 3853014, PROT_READ, MAP_PRIVATE|MAP_LOCKED, 108, 0) = 0x7f67a9b27000
close(108)                              = 0
madvise(0x7f67a9b27000, 3853014, MADV_WILLNEED) = 0
madvise(0x7f67a9b27000, 3853014, MADV_RANDOM) = 0
sysinfo({uptime=9261, loads=[354464, 197600, 128576], totalram=67231506432, freeram=46524735488, sharedram=5890048, bufferram=46723072, totalswap=1027600384, freeswap=1027600384, procs=258, totalhigh=0, freehigh=0, mem_unit=1}) = 0
openat(AT_FDCWD, "/opt/versa/etc/spack/installed/current/database/geoip/vs_geoip.json", O_RDONLY) = 108
fstat(108, {st_mode=S_IFREG|0644, st_size=28684458, ...}) = 0
mmap(NULL, 28684458, PROT_READ, MAP_PRIVATE, 108, 0) = 0x7f67a7fcb000
close(108)                              = 0
mmap(NULL, 50331648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f67a4fcb000
mmap(NULL, 58720256, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f67a17cb000
madvise(0x7f67a544b000, 1314816, MADV_DONTNEED) = 0
mmap(NULL, 67108864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f679d7cb000
mmap(NULL, 83886080, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f6797a00000
mmap(NULL, 33554432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f6795a00000
madvise(0x7fa860201000, 10489856, MADV_DONTNEED) = 0
madvise(0x7f67a5f0e000, 2625536, MADV_DONTNEED) = 0
mmap(NULL, 100663296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f678fa00000
madvise(0x7f67a75cf000, 5246976, MADV_DONTNEED) = 0
munmap(0x7f67a7fcb000, 28684458)        = 0
mmap(NULL, 117440512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f6788a00000
^Cstrace: Process 11225 detached

[mangod9]

Does it only repro when published as NativeAOT binary?

[FilipVD]

It's a library. Uless I'm mistaken only .net AOT can be called from C++ code. So a library in .Net (non AOT) could not work in this setup.

[jkotas]

Unless I'm mistaken only .net AOT can be called from C++ code

You can use https://github.com/AaronRobinsonMSFT/DNNE to publish a library that uses regular CoreCLR w/ JIT. However, I doubt that it is going to make a difference. If anything, I expect that we will see the exception sooner with regular CoreCLR. It would be useful data point to confirm it.

[landerverhacklansweeper]

You can use https://github.com/AaronRobinsonMSFT/DNNE to publish a library that uses regular CoreCLR w/ JIT. However, I doubt that it is going to make a difference. If anything, I expect that we will see the exception sooner with regular CoreCLR. It would be useful data point to confirm it.

Thank you for this input, I didn't know about this option.
Before trying this with DPDK, I tried running the sample from DNNE in Ubuntu 18.04. It didn't work with both .NET6.0 and .NET8.0.

Calling managed export
Aborted (core dumped)

[New LWP 18333]
Dwarf Error: Cannot handle DW_FORM_<unknown> in DWARF reader [in module ./sample_native.so]
Core was generated by `./main'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

It does work (.NET8.0) in Ubuntu 24.04 though. Even when the .so is compiled in 18.04, I can run it on 24.04. Are there any restrictions for DNNE and older versions of Ubuntu?
We are primarily concerned with Ubuntu 18.04.

Also I seems like DNNE ignores the --self-contained flag? (which doesn't matter if we just use DNNE during troubleshooting)

[jkotas]

Ubuntu 18.04

Ubuntu 18.04 standard support ended in 2023. It is not surprising that the latest bits do not work well on it.

We are not testing .NET 8 runtime on Ubuntu 18.04 either. The list of Ubuntu versions that .NET 8 runtime is tested and supported on is in https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#linux .

What prevents you from moving to newer version?

[landerverhacklansweeper]

I would love to move to a newer version. But we are working with networking devices that are already deployed with many customers. The hardware and OS are closely aligned, and can't be updated. We can only update the OS when shipping new devices.
I know it's unfortunate.
In the not so distant future, we can targeting Ubuntu 22. But not today.

[landerverhacklansweeper]

The Stacktrace stops after this:

Unhandled Exception: System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Threading.Thread.StartCore() + 0xa5
   at System.Threading.PortableThreadPool.GateThread.CreateGateThread() + 0x92

When inspecting the code I can see that the C# code calls some C++ code. that fails.
I can trace down to the ThreadNative_Start function. But I'm lost after that. Is there a way to get a trace on this level? within the C++ code?

Any help is much appreciated

[jkotas]

The sources for native AOT are here:

runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs

Line 388 in c0c7f02

throw new OutOfMemoryException();

It calls this:

runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.Unix.cs

Lines 97 to 100 in c0c7f02

	if (!Interop.Sys.CreateThread((IntPtr)_startHelper!._maxStackSize, &ThreadEntryPoint, GCHandle<Thread>.ToIntPtr(thisThreadHandle)))
	{
	return false;
	}

And then this:

runtime/src/native/libs/System.Native/pal_threading.c

Lines 219 to 264 in c0c7f02

	int32_t SystemNative_CreateThread(uintptr_t stackSize, void *(*startAddress)(void*), void *parameter)
	{
	bool result = false;
	pthread_attr_t attrs;
	int error = pthread_attr_init(&attrs);
	if (error != 0)
	{
	// Do not call pthread_attr_destroy
	return false;
	}
	error = pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED);
	assert(error == 0);
	#ifdef HOST_APPLE
	// Match Windows stack size
	if (stackSize == 0)
	{
	stackSize = 1536 * 1024;
	}
	#endif
	if (stackSize > 0)
	{
	if (stackSize < (uintptr_t)PTHREAD_STACK_MIN)
	{
	stackSize = (uintptr_t)PTHREAD_STACK_MIN;
	}
	error = pthread_attr_setstacksize(&attrs, stackSize);
	if (error != 0) goto CreateThreadExit;
	}
	pthread_t threadId;
	error = pthread_create(&threadId, &attrs, startAddress, parameter);
	if (error != 0) goto CreateThreadExit;
	result = true;
	CreateThreadExit:
	error = pthread_attr_destroy(&attrs);
	assert(error == 0);
	return result;
	}

My guess is that pthread_create is failing

[landerverhacklansweeper]

@jkotas Thank you. I was looking in the wrong place it seems.
So eventually it calls the pthread_create which is in a separate library (not part of the runtime). I would love to see which parameters are passed to this function. I'm not used to working in this environment. How could I log this?

[jkotas]

Are you able to run the app under native debugger (gdb or lldb)?

If you can run under native debugger, you can set a breakpoint at pthread_create method and inspect the arguments when the breakpoints gets hit.

If you cannot run under native debugger, you would have to build a private runtime with ad-hoc logging for the arguments you are interested in.