Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected ObjectDisposedException when IoT Hub is throttling #6042

Closed
danigian opened this issue Jan 25, 2022 · 13 comments
Closed

Unexpected ObjectDisposedException when IoT Hub is throttling #6042

danigian opened this issue Jan 25, 2022 · 13 comments
Assignees
@varunpuranik @ancaantochi
Labels
customer-reported iotedge no-issue-activity

Comments

@danigian
Copy link

Expected vs current behavior

Assuming a IoT Hub with 100 devices, one edge gateway and one module with the following code

// Creating registry manager in order to get 50 connection strings
var registry = RegistryManager.CreateFromConnectionString(iothubownerstring);

var deviceTwins = await registry.CreateQuery("select * from devices", 50).GetNextAsTwinAsync();

// Getting 50 devices
var listDeviceTasks = new List<Task<Device>>();
foreach (var element in deviceTwins)
{
	listDeviceTasks.Add(registry.GetDeviceAsync(element.DeviceId));
}
var devices = await Task.WhenAll(listDeviceTasks);

// Building up the connection string manually
var connectionStrings = new List<string>();
foreach (var d in devices)
{
	connectionStrings.Add($"HostName={iothubname}.azure-devices.net;DeviceId={d.Id};SharedAccessKey={d.Authentication.SymmetricKey.PrimaryKey};GatewayHostName={gwhostname}");
}

// Setting AMQP Transport Settings in such a way that Operation Timeout is 10 seconds instead of default 60secs
var amqp = new AmqpTransportSettings(Microsoft.Azure.Devices.Client.TransportType.Amqp_Tcp_Only)
{
	AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
	{
		Pooling = true,
		MaxPoolSize = 1
	},
	OperationTimeout = TimeSpan.FromSeconds(3)
};
amqp.RemoteCertificateValidationCallback += (_, _, _, _) => true;

// Constructing list of GetTwin Task
var tasks = new List<Task<Twin>>();
foreach (var element in connectionStrings)
{
	var client = DeviceClient.CreateFromConnectionString(element, new[] { amqp });

	client.SetRetryPolicy(new ExponentialBackoff(int.MaxValue,
				minBackoff: TimeSpan.FromMilliseconds(100),
				maxBackoff: TimeSpan.FromSeconds(10),
				deltaBackoff: TimeSpan.FromMilliseconds(100)));
	tasks.Add(client.GetTwinAsync());
}

await Task.WhenAll(tasks);

I would expect that, for a S1 IoT Hub, the first time I am awaiting the last "tasks", I should be throttled by the amount of registry operations allowed in one single minute.

Therefore I would either expect something to be thrown concerning the Throttling or, given the OperationTimeout set to 3 seconds for AMQP transport setting, a TimeoutException.

Instead, what really happens is that a ObjectDisposedException is thrown. See details below

Exception stacktrace 
System.ObjectDisposedException:
   at Microsoft.Azure.Devices.Client.AuthenticationWithTokenRefresh+<GetTokenAsync>d__19.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsTokenProvider+<GetTokenAsync>d__3.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.TaskHelpers.EndAsyncResult (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.IteratorAsyncResult`1.StepCallback (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.AsyncResult.End (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.AmqpCbsLink+<>c__DisplayClass4_0.<SendTokenAsync>b__1 (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsLink+<SendTokenAsync>d__2.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpAuthenticationRefresher+<InitLoopAsync>d__10.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotConnection+<CreateRefresherAsync>d__9.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder+<CreateRefresherAsync>d__14.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<EnsureSessionIsOpenAsync>d__28.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<OpenAsync>d__27.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler+<OpenAsync>d__13.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler+<OpenAsync>d__7.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<>c__DisplayClass27_0+<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<ExecuteWithErrorHandlingAsync>d__28`1.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<OpenInternalAsync>d__38.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<EnsureOpenedAsync>d__36.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<>c__DisplayClass29_0+<<SendTwinGetAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<SendTwinGetAsync>d__29.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at TestCode.DeviceClient+<GetTwinAsync>d__8.MoveNext (TestCode.DeviceClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /TestCode/DeviceClient.cs:74)

Context (Environment)

support_bundle.zip file here
@danigian danigian changed the title Unexpected ObjectDisposedException Unexpected ObjectDisposedException when IoT Hub is throttling Jan 25, 2022
@scb01 scb01 self-assigned this Jan 26, 2022
@varunpuranik
Copy link
Contributor

@danigian - this seems like an issue with the C# SDK than with IoT Edge. Do you get the same error if you connect to IoT Hub instead of the IoT Edge gateway (by not adding the GatewayHostName to the connection string)?

@drwill-ms @azabbasi as fyi

@drwill-ms
Copy link
Contributor

I agree this likely belongs over in https://github.com/Azure/azure-iot-sdk-csharp/issues instead. I can't initiate a transfer though. Can you @varunpuranik?

I'm not sure what is being disposed here. None of the sample code has a call to Dispose nor any using keywords which would cause disposal.

Indeed, this call stack doesn't show that we even got to the "get twin" stage; it is just trying to connect.

@varunpuranik
Copy link
Contributor

Yeah, I am not able to transfer it to the azure-iot-sdk-csharp repo either. @danigian - can you please open the issue in the csharp SDK repo and close this one? Thanks!

@danigian
Copy link
Author

@varunpuranik @drwill-ms I previously did the test with C# SDK 1.39.0, but I have done it again now with 1.40.0.
The issue is reproducing only with the GatewayHostName added to the connection string.
If I am not going through edgeHub the issue is not reproducing, therefore I am not sure we should transfer the issue to the azure-iot-sdk-csharp repository.

As additional input, I am using CloudAndScope Authentication mode in edgeHub and devices are not children of any edge device in my IoT Hub

@varunpuranik
Copy link
Contributor

@danigian - Thanks for the additional information - in that case it seems like something in EdgeHub's behavior is causing the SDK instance to go into a bad state.
We will have to try to reproduce it in that case.
One question - any reason you need to use CloudAndScope authentication (as opposed to Scope authentication, as is recommended)?

@danigian
Copy link
Author

@varunpuranik, thanks for your availability in trying to reproduce the issue.
I am working on the Azure IoT Edge LoRaWAN Starter Kit
 for which we already had a discussion regarding usage of CloudAndScope authentication requirements (mainly due to roaming devices that should not be strictly linked to one edge gateway)
If you want, I am available to expand on this point.

@scb01
Copy link
Contributor

scb01 commented Feb 23, 2022

Assigning issue to @varunpuranik to follow up.

@scb01 scb01 assigned varunpuranik and unassigned scb01 Feb 23, 2022
@danigian
Copy link
Author

danigian commented Mar 1, 2022

@varunpuranik any update on this?

@github-actions
Copy link

github-actions bot commented Apr 1, 2022

This issue is being marked as stale because it has been open for 30 days with no activity.

@ancaantochi
Copy link
Contributor

@danigian

I was able to reproduce the issue, it is the authentication which causes it because the devices are not in scope for edge device and it uses CloudAndScope. Because the authentication takes longer, the SDK doesn't handle it properly and I would recommend to open an issue with them.
There is a workaround that you could use is to first call OpenAsync and await so authentication is already completed by the time you call GetTwinAsync.

@ancaantochi ancaantochi self-assigned this Apr 6, 2022
@danigian
Copy link
Author

danigian commented Apr 7, 2022

Hi @ancaantochi, thanks for your explanation.
Could you provide some more details in order to open the issue on the C# SDK side?

@drwill-ms
Copy link
Contributor

@danigian you can post it here.

@jlian
Copy link
Member

jlian commented Apr 12, 2022

Thanks @drwill-ms. @danigian closing this issue here to continue the investigation via the C# repo

@jlian jlian closed this as completed Apr 12, 2022
@danigian danigian mentioned this issue Apr 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@varunpuranik varunpuranik

@ancaantochi ancaantochi

Labels
customer-reported iotedge no-issue-activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@danigian @jlian @scb01 @varunpuranik @drwill-ms @ancaantochi