When EdgeHub disconnects from IoT Hub, Desired Properties are being lost #1654
Comments
|
@grzegorski Can you attach the full logs (drag and drop the files into a comment here) for edge hub and edge agent? The excerpt above shows that edge hub is having difficulty connecting (presumably because the network is unstable), but we need more context. It would also be great to get more verbose debug logs for edge hub, but that would require reproducing the scenario again, and the log files would be a lot bigger. Finally, when you say your module loses desired properties, is this the scenario?
|
|
We have experienced the same issue in a few of our running modules (not all of them) last weekend. |
arsing
added
the
needinfo
|
The same has been observed again on other IoT Edge Device. Our module lost all the desired properties and stopped processing messages after EdgeHub lost connection with the cloud (over AMQP WebSocets). Another site this morning: <6> 2019-09-05 07:07:59.051 +00:00 [INF] - Retrying cloud proxy operation UpdateReportedPropertiesAsync for 1002-kilbourn-iotedge-01/$edgeHub. Custom Module Logs: 2019-09-05 07:04:06.793 +00:00 [WRN]: Value for LOG_LEVEL not found. Using defaultValue [information] Unhandled Exception: System.AggregateException: One or more errors occurred. (The CancellationTokenSource has been disposed.) ---> System.ObjectDisposedException: The CancellationTokenSource has been disposed. |
|
@grzegorski I'm looking at the logs right now. Question: what version of the device SDK (Microsoft.Azure.Devices.Client) are you using in each of your modules (iotHubRelay, opcMapper, opcPublisher, tmgHeartbeat)? |
|
Looking at the last logs you uploaded (edgeAgent.log, edgeHub.log, opcMapper.log), I don't see any obvious problems in Edge Hub. Here's a timeline of events in the log, as far as I can tell:
It would be good to address the ObjectDisposedException in opcMapper, but I'm not convinced that's contributing to your problem. Are these warnings in opcMapper.log the reason why you say desired properties are being lost? |
|
We are also facing same issue ... so is there any update on this fix ?? |
|
This issue is being marked as stale because it has been open for 30 days with no activity. |
|
@varunpuranik can you please take a look at this one? |
|
Based on the information in this issue, I don't think edge hub is actually losing desired properties. Rather I think the custom module ( Specifically, the callback that receives incremental updates to the twin desired properties is only useful if you already have the previous version of the twin to compare against. So if you ever disconnect/reconnect, or if you discover a discontinuity in the twin version field, you need to get the full twin again. A great example of this can be found in our Edge Agent. The callback function that we register with the SDK (by calling In general, the algorithm we use to keep an up-to-date copy of desired properties is:
The file EdgeAgentConnection.cs has nearly all the code to do this, and is fairly well organized. You should be able to adapt it to your needs pretty easily. |
|
@damonbarry We observed issue today again. Our modules handle now the desired properties update and they didn't fail but EdgeHub and Edge Agent failed to update desired properties and stopped sending anything to the cloud. At the same time IoT Hub reported all connections to be OK. |
|
@grzegorski Thanks for opening #1974; we can track this latest instance in that issue. |
|
Closing for now, using #1974 to track, please re-open as needed. |
Expected Behavior
IoT Edge device is running in some locations where upstream connectivity can be challenging.
Our module reads configuration from desired properties.
Expected behavior is that when EdgeHub losses cloud connectivity for whatever reason. Desired Properties for all modules are maintained and refreshed when the connectivity is resumed.
Current Behavior
We noticed that Desired Properties are lost occasionally, normally after network connectivity issues , causing modules to loose parameters that are read from there.
Steps to Reproduce
1.Deploy IoT Edge modules with upstream connectivity over Web Sockets
2.When the cloud connection over Web Sockets fails, UpdateReportedPropertiesAsync method times out and module loses Desired Properties and doesn't recover
Device (Host) Operating System
Ubuntu Server 18.04
Architecture
amd64
Runtime Versions
iotedged
1.0.8
Edge Agent
1.0.8
Edge Hub
1.0.8
Docker
3.0.6
Client:
Version: 3.0.6
API version: 1.40
Go version: go1.12.5
Git commit: a63faebc
Built: Wed Jun 19 22:56:37 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 3.0.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.1
Git commit: dbe4a30
Built: Thu Apr 18 22:07:58 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.5
GitCommit: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc:
Version: 1.0.0-rc6+dev
GitCommit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
docker-init:
Version: 0.18.0
GitCommit: fec3683
Logs
module logs
edge-agent logs
edge-hub logs
Additional Information
The text was updated successfully, but these errors were encountered: