SetMethodHandlerAsync throws SemaphoreFullException

[arabelaa]

Context

• OS, version, SKU and CPU architecture used: x64, Windows 10
• Application's .NET Target Framework : netcoreapp3.1
• Device: Intel NUC
• SDK version used: 1.26.0

Description of the issue

Exception is sometimes thrown when registering direct methods. This doesn't happen all the time. I am afraid something is wrong in a different thread causing the executing thread to release the semaphore twice. I can't reproduce the issue locally, unfortunately.

 private async Task RegisterDirectMethodsAsync()
        {
           //_directMethodRegistrations is a dictionary that contains the method name and the callback 
            await Task.WhenAll(_directMethodRegistrations.Registrations.Select(pair =>
                _deviceClient.SetMethodHandlerAsync(pair.Key, pair.Value, null)));

            await _deviceClient.SetMethodDefaultHandlerAsync(OnDefaultHandlerCalled, null);
        }

System.Threading.SemaphoreFullException:
   at System.Threading.SemaphoreSlim.Release (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.InternalClient+<SetMethodHandlerAsync>d__74.MoveNext (Microsoft.Azure.Devices.Client, Version=1.23.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.InternalClient+<SetMethodHandlerAsync>d__73.MoveNext (Microsoft.Azure.Devices.Client, Version=1.23.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   a

[timtay-microsoft]

Can you describe roughly how you are using the device client around the times that you have seen it? Are you using the device client in only one thread? Are you also setting up any other callbacks on the SDK? And is the device client open before you call SetMethodHandlerAsync?

[arabelaa]

I have the DeviceClient registered as singleton in DI (Microsoft DI).
Beside the directmethods I am also register a twinchange callback.
await _deviceClient.SetDesiredPropertyUpdateCallbackAsync(OnDesiredPropertyChangedAsync, null); I first register the directmethods and afterwards the twinchange.

I am initializing the deviceClient using Microsoft.Azure.Devices.Client.DeviceClient.CreateFromConnectionString() . I don't explicity open the deviceClient to register the callbacks.

I can imagine it's hard to pinpoint the source of the problem.

[timtay-microsoft]

I have the DeviceClient registered as singleton in DI (Microsoft DI).

Are you using this device client singleton in multiple threads at once, though?

[arabelaa]

There are multiple places where the deviceclient is used (for listening and sending events) - so it can be that multiple threads are accessing the deviceclient at once.
But when I register the type inside the DI it means that the service is thread safe

The type that I am registering is a wrapper around the Microsoft.Azure.Devices.Client.DeviceClient - and in the constructor I do something like

public class DeviceClient : IDeviceClient
{
    private readonly Microsoft.Azure.Devices.Client.DeviceClient _deviceClient;

    public DeviceClient(ISettings settings)
        {
            _deviceClient = Microsoft.Azure.Devices.Client.DeviceClient.CreateFromConnectionString(settings
                .GetConnectionString().GetAwaiter().GetResult());
            _deviceClient.OperationTimeoutInMilliseconds = (uint) TimeSpan.FromMinutes(1).TotalMilliseconds;
        }
}

// and in the ConfigureServices I have
services.AddSingleton<IDeviceClient, DeviceClient>();

[timtay-microsoft]

I believe we have identified the fix for this issue. There were some instances where our code would fail to grab the semaphore but still try to release it anyways. The fix has now been checked in and we'll update this thread once it has been released.

[abhipsaMisra]

This fix has now been released: https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/lts_2020-8-19

[az-iot-builder-01]

@timtay-microsoft, @arabelaa, @abhipsaMisra, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey