Linux-arm64, SemaphoreSlim Release does not always work as expected - possible race condition

[thefex]

Description

The bug is related to: linux-arm64 builds but might not be limited to this architecture. I tried osx-arm64 but I could not reproduce it there.

When using SemaphoreSlim .WaitAsync() and then Release() - there are cases where Release() executes but it does not properly increment Semaphore counter.

Please take a look at stripped logcat logs that reproduce the issue (attachments .log) with similar sample as below.

Sample method that breaks:

public static readonly SemaphoreSlim _semaphore = new SemaphoreSlim(1);
public static bool someCondition = true;
public async Task TestMethod()
{
    var uuid = Guid.NewGuid().ToString("d");
    try
    {
        _logger.LogInformation("Before Semaphore Wait: " + uuid + ", counter: " + _semaphore.CurrentCount);
        await _semaphore.WaitAsync();
        _logger.LogInformation("After Semaphore Wait: " + uuid + ", counter: " + _semaphore.CurrentCount);

        if (someCondition)
        {
            // here in real sample, is a long running task, that can potentially switch threads (unsure about that though)
            await Task.Run(async () =>
            {
                // simulate longer running task
                await Task.Delay(50);
            });
            
            someCondition = false;
        }
    }
    finally
    {
        _logger.LogInformation("Releasing Semaphore: " + uuid + ", counter: " + _semaphore.CurrentCount);
        _semaphore.Release();
        for (int i = 0; i < 10; ++i)
        {
            await Task.Delay(100);
            _logger.LogInformation("After Semaphore Release: " + uuid + ", counter: " + _semaphore.CurrentCount);
        }
    }
}

And this leads to Semaphore keeping "0" counter even thought it was properly released twice! (see: logs_semaphoreslim_bug_01.log)

However, if we add: "Task.Delay(5)" just after _semaphore.WaitAsync() - we get valid "1" counter value post release (see: logs_semaphoreslim_with_task_delay_5_bug_02).

Also, if I replaced SemaphoreSlim with Semaphore and changed _semaphore.WaitAsync() with "await Task.Run(() => semaphore.WaitOne());" - the bug is not reproducible anymore.

Suggesting there is some nasty race condition in SemaphoreSlim linux threading implementation.

The overall project runs on Android (arm64 device), NativeAOT builds .so file through linux-bionic-arm64 architecture that is then loaded in Android project. Android project calls FFI exported via UnmanagedCallersAttribute - which properly calls the C# methods.

Reproduction Steps

Please follow up with defect description.

Expected behavior

After SemaphoreSlim Release count should increase to 1.

Actual behavior

When SemaphoreSlim Release called immediately one after another - count stay at 0 suggesting some nasty race condition in runtime.

Regression?

No, exists both .NET9/10.

Known Workarounds

Don't use SemaphoreSlim - the issue does not exists when SemaphoreSlim is replaced with Semaphore as:

await Task.Run(() => semaphore.WaitOne())
..
semaphore.Release()

Or...
add nasty await Task.Delay(5) after semaphore.WaitAsync()

Configuration

.NET: tested on both: .NET 10.0.101.1 and .NET 9.0.302 - both bugged.
Build OS: OSX 15.7
Architecture: reproducing on linux-bionic-arm64 - not sure if that's arm64 specific
Test setup:
Library compiled as .so through NativeAOT and then used in Android/Kotlin project

Other information

logs_semaphoreslim_bug_01.log
logs_semaphoreslim_with_task_delay_5_bug_02.log

[thefex]

pinging @MichalStrehovsky as it seems you know the most about native-aot area

[jkotas]

it seems you know the most about native-aot area

This bug does not look nativeaot-specific.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[thefex]

it seems you know the most about native-aot area

This bug does not look nativeaot-specific.

This does not happen in classic .NET/JIT or even mono/monoaot runtime though. It looks linux-bionic (maybe arm64 only?) specific

[jkotas]

This does not happen in classic .NET/JIT or even mono/monoaot runtime though.

It is not unusual for race conditions to repro in a specific configuration only, even though they affect all configurations.

SemaphoreSlim is 100% portable code. It does not have any runtime-specific code. If the issue was nativeaot specific, it would have to be several layers below SemaphoreSlim that is less likely.

Any chance you can produce a standalone repro? I doubt that we will be able to diagnose this from the Android log.

[thefex]

This does not happen in classic .NET/JIT or even mono/monoaot runtime though.

It is not unusual for race conditions to repro in a specific configuration only, even though they affect all configurations.

SemaphoreSlim is 100% portable code. It does not have any runtime-specific code. If the issue was nativeaot specific, it would have to be several layers below SemaphoreSlim that is less likely.

Any chance you can produce a standalone repro? I doubt that we will be able to diagnose this from the Android log.

I see. No I dont think I am able to do so. I am able to reproduce it on constant basis though - so if someone on NativeAOT could guide me what kind of disassembly (and potentialy how to get that) can help

The clue (or rather guess) I could see is that JIT might insert additional instructions, nativeaot could emit different instruction set and that might be arm64 specific for ex. In reading this volatile variable.

Also the threading part is working differently in naot builds as far as I know

[stephentoub]

And this leads to Semaphore keeping "0" counter even thought it was properly released twice! (see: logs_semaphoreslim_bug_01.log)

If there's a pending wait on the semaphore when release is called, the count won't be increased; it'll instead transfer ownership of that released "count" to the pending waiter, so the count will remain zero. Are you sure that's not what you're witnessing?

[thefex]

And this leads to Semaphore keeping "0" counter even thought it was properly released twice! (see: logs_semaphoreslim_bug_01.log)

If there's a pending wait on the semaphore when release is called, the count won't be increased; it'll instead transfer ownership of that released "count" to the pending waiter, so the count will remain zero. Are you sure that's not what you're witnessing?

The thing is that following code start to work when replaced with Semaphore and WaitOne() - in nativeaot context. In the SemaphoreSlim scenario - it just blocks forever.

Also - the code itself (SemaphoreSlim) works alright when deployed through net-android/net-ios workflow (non nativeaot env).

We are exporting some kind of enterprise SDK (already in production for .net-mobile) with custom in-house tooling that auto-generate us native android/ios (kotlin/swift) code - that just calls/marshal data through [UnmanagedCallers..] native-aot exported methods.

[alexTr3]

this is just an awfull bug. should be priority #1

after review by stephen it seem I had the waitasync inside the try/finally block causing my intermittent problems.. sorry for that

[stephentoub]

@alexTr3, can you elaborate? You have a separate case of the same thing from @thefex, same environment, etc? Can you share a minimal repro?