Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error while refreshing the service identity #6936

Closed
adcoly opened this issue Feb 28, 2023 · 22 comments
Closed

Error while refreshing the service identity #6936

adcoly opened this issue Feb 28, 2023 · 22 comments
Assignees
@nyanzebra
Labels
customer-reported iotedge

Comments

@adcoly
Copy link

adcoly commented Feb 28, 2023

Expected Behavior

I shouldn't have connection issue when the edgeHub try to refresh the service identity for my Gateway (LBS)

Current Behavior

I meet issue on refreshing the service identity for my Gateway (see logs below). I have a "System.TimeoutException: Operation timed out" error.

Steps to Reproduce

I do not have specific steps to reproduce to head to this issue.

Context (Environment)

Output of iotedge check

Click here 

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
× aziot-identity-service package is up-to-date - Error
    could not query https://aka.ms/latest-aziot-identity-service for latest available version
‼ host time is close to reference time - Warning
    Could not query NTP server
√ preloaded certificates are valid - OK
× keyd is running - Error
    Could not connect to keyd on unix:///run/aziot/keyd.sock
× certd is running - Error
    Could not connect to certd on unix:///run/aziot/certd.sock
× identityd is running - Error
    Could not connect to identityd on unix:///run/aziot/identityd.sock

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.    Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
    docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded.
    See 'docker run --help'.
× aziot-edge package is up-to-date - Error
    Error while fetching latest versions of edge components: could not send HTTP request
× container time is close to host time - Error
    Could not query local time inside container
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
12 check(s) succeeded.
8 check(s) raised warnings. Re-run with --verbose for more details.
7 check(s) raised errors. Re-run with --verbose for more details.
11 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

  • Host OS: Ubuntu 20.04
  • Architecture: amd64
  • Container OS: Linux containers

Runtime Versions

  • aziot-edged [run iotedge version]: 1.4.9
  • Edge Agent [image tag (e.g. 1.0.0)]: 1.4.2
  • Edge Hub [image tag (e.g. 1.0.0)]: 1.4.2
  • Docker/Moby [run docker version]:
Click here 

Client:
 Version:           20.10.23+azure-2
 API version:       1.41
 Go version:        go1.19.6
 Git commit:        715524332ff91d0f9ec5ab2ec95f051456ed1dba
 Built:             Wed Jan 18 20:42:16 UTC 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.22+azure-1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b314993e5eb3cc2776da0bbe41d5eb4b707b
  Built:            Thu Dec 15 22:17:04 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18+azure-1
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:

Logs

Logs for the last 30min:

Additional Information

I had never saw this issue with those runtime versions:

  • aziot-edged [run iotedge version]: 1.4.9
  • Edge Agent [image tag (e.g. 1.0.0)]: 1.1
  • Edge Hub [image tag (e.g. 1.0.0)]: 1.1
@nyanzebra nyanzebra self-assigned this Feb 28, 2023
@nyanzebra
Copy link
Contributor

Hi @adcoly, have a few questions to help understand the problem more :)

Is this something that was not a problem with 1.1, then you upgraded and started seeing the issue? Also is this a prod environment?

Would you mind also sharing the output of the following:
sudo systemctl status aziot-identityd
sudo systemctl status aziot-keyd
sudo systemctl status aziot-certd

Example output:
aziot-identityd.service - Azure IoT Identity Service
Loaded: loaded (/lib/systemd/system/aziot-identityd.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2023-02-28 18:27:49 UTC; 4h 13min ago
TriggeredBy: ● aziot-identityd.socket
Main PID: 22593 (aziot-identityd)
Tasks: 8 (limit: 19192)
Memory: 2.8M
CGroup: /system.slice/aziot-identityd.service
└─22593 /usr/libexec/aziot-identity-service/aziot-identityd

@adcoly
Copy link
Author

adcoly commented Mar 1, 2023 •
edited

As far as I remember/noticed, it was not a problem when aziot-edged was in 1.4.9 and edgeAgent/edgeHub in 1.1.
It is an indus environment.

The 3 services were stopped since 5 days. We restarted it few minutes before sharing you status:

  • aziot-identityd
● aziot-identityd.service - Azure IoT Identity Service
     Loaded: loaded (/lib/systemd/system/aziot-identityd.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-01 08:04:28 UTC; 1h 37min ago
TriggeredBy: ● aziot-identityd.socket
   Main PID: 3884456 (aziot-identityd)
      Tasks: 8 (limit: 7022)
     Memory: 12.1M
     CGroup: /system.slice/aziot-identityd.service
             └─3884456 /usr/libexec/aziot-identity-service/aziot-identityd
  • aziot-keyd
● aziot-keyd.service - Azure IoT Keys Service
     Loaded: loaded (/lib/systemd/system/aziot-keyd.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-01 08:20:02 UTC; 1h 21min ago
TriggeredBy: ● aziot-keyd.socket
   Main PID: 3888765 (aziot-keyd)
      Tasks: 8 (limit: 7022)
     Memory: 5.2M
     CGroup: /system.slice/aziot-keyd.service
             └─3888765 /usr/libexec/aziot-identity-service/aziot-keyd
  • aziot-certd
● aziot-certd.service - Azure IoT Certificates Service
     Loaded: loaded (/lib/systemd/system/aziot-certd.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-01 08:21:59 UTC; 1h 18min ago
TriggeredBy: ● aziot-certd.socket
   Main PID: 3926410 (aziot-certd)
      Tasks: 8 (limit: 7022)
     Memory: 3.8M
     CGroup: /system.slice/aziot-certd.service
             └─3926410 /usr/libexec/aziot-identity-service/aziot-certd

By the way, we noticed on this page that the identity service is now resposible of components identity that communicates with the IoTHub. We are currently verifying our configuration for this specific service. It seems it is configured through the /etc/aziot/config.toml configuration file.
Here is what we have in it for the moment:

[provisioning]
iothub_hostname = "xxxxxxxxxxxx.azure-devices.net"
source = "dps"
global_endpoint = "https://xxxxxxxxxxxx.azure-devices-provisioning.net"
id_scope = "xxxxxxxxxxxx"

[provisioning.attestation]
method = "symmetric_key"
registration_id = "xxxxxxxxxxxx"
symmetric_key = { value = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }

trust_bundle_cert = "file:///var/opt/ssl/xxxxxxxxxxxx.pem"
cert = "file:///var/opt/ssl/xxxxxxxxxxxx.pem"
key = "file:///var/opt/ssl/xxxxxxxxxxxx.key"

auto_reprovisioning_mode = "Dynamic
"auto_generated_edge_ca_expiry_days = 90

[agent]
name = "edgeAgent"
ype = "docker"

[agent.config]
image = "xxxxxxxxxxxx.com/azureiotedge-agent:1.4"

@nyanzebra
Copy link
Contributor

@adcoly so edgeHub will not be able to be connected when keyd is not up, which based on above iotedge check that seemed to have been the case. Though from the checks I asked you to run it appears to be running now. Not sure what caused those daemons to not be working though...

Is this still an ongoing issue for you? Also, any idea on what might have been going on that might have caused those daemons to not be up and running? Any changes to this particular device? Or just from refreshing service gateway?

If this is reproducible, it will be great if you can get logs for keyd, certd, and identityd too :) Curious to know what happened

@adcoly
Copy link
Author

adcoly commented Mar 3, 2023 •
edited

We still have some issues in the edgeHub.

edgeHub logs here ``` <6> 2023-03-03 09:33:57.710 +00:00 [INF] - Entering periodic task to reauthenticate connected clients <6> 2023-03-03 09:33:57.711 +00:00 [INF] - Unable to authenticate client 00137A1000011656 with cached service identity 00137A1000011656 (Found: False). Resyncing service identity... <6> 2023-03-03 09:33:57.711 +00:00 [INF] - Refreshing service identity for 00137A1000011656 <4> 2023-03-03 09:33:58.181 +00:00 [WRN] - Error while refreshing the service identity: 00137A1000011656 OnBehalfOf: carsi4iiotedge1 Microsoft.Azure.Devices.Edge.Hub.Core.DeviceInvalidStateException: Device not in scope: [Unauthorized: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:0f8522d5e91e48f98998ba9a112f14fa-G:0-TimeStamp:03/03/2023 09:33:58"}]. ---> Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceScopeApiException: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:0f8522d5e91e48f98998ba9a112f14fa-G:0-TimeStamp:03/03/2023 09:33:58"} at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfInternalAsync(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 191 at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 137 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfWithRetry(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 140 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 at Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeIdentitiesCache.RefreshServiceIdentityInternal(String refreshTarget, String onBehalfOfDevice, Boolean invokeServiceIdentitiesUpdated) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/DeviceScopeIdentitiesCache.cs:line 201 <6> 2023-03-03 09:33:58.182 +00:00 [INF] - Removed node: 00137A1000011656 <6> 2023-03-03 09:33:58.183 +00:00 [INF] - Device scope identities refresh is ready because a refresh was signalled. <6> 2023-03-03 09:33:58.183 +00:00 [INF] - Starting refresh of device scope identities cache <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1 <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/$edgeAgent <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/$edgeHub <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/GenericDecoder <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/MetadataEnricher <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/NkeSensorsDecoder <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/NetvoxSensorsDecoder <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/NKESensorsDecoderPy <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/Kafka2Pi <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/LoRaWanNetworkSrvModule <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/metricbeat <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/filebeat <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Not changed node: carsi4iiotedge1/DefenderIotMicroAgent <6> 2023-03-03 09:34:10.600 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes. <6> 2023-03-03 09:34:10.601 +00:00 [INF] - Unable to authenticate client 70B3D5E75E00CA92 with cached service identity 70B3D5E75E00CA92 (Found: False). Resyncing service identity... <6> 2023-03-03 09:34:10.601 +00:00 [INF] - Refreshing service identity for 70B3D5E75E00CA92 <4> 2023-03-03 09:34:10.780 +00:00 [WRN] - Error while refreshing the service identity: 70B3D5E75E00CA92 OnBehalfOf: carsi4iiotedge1 Microsoft.Azure.Devices.Edge.Hub.Core.DeviceInvalidStateException: Device not in scope: [Unauthorized: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:7c25ef70e08142cd80aadc0c4cbc5012-G:0-TimeStamp:03/03/2023 09:34:10"}]. ---> Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceScopeApiException: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:7c25ef70e08142cd80aadc0c4cbc5012-G:0-TimeStamp:03/03/2023 09:34:10"} at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfInternalAsync(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 191 at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 137 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfWithRetry(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 140 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 at Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeIdentitiesCache.RefreshServiceIdentityInternal(String refreshTarget, String onBehalfOfDevice, Boolean invokeServiceIdentitiesUpdated) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/DeviceScopeIdentitiesCache.cs:line 201 <6> 2023-03-03 09:34:10.781 +00:00 [INF] - Removed node: 70B3D5E75E00CA92 <6> 2023-03-03 09:34:10.782 +00:00 [INF] - Skipping cache refresh, waiting 119 seconds until refreshing again. <6> 2023-03-03 09:34:12.584 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:34:12.586 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 2.3684ms" <6> 2023-03-03 09:34:42.583 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:34:42.585 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 2.0757ms" <6> 2023-03-03 09:35:12.583 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:35:12.585 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 1.7187ms" <6> 2023-03-03 09:35:42.585 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:35:42.587 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 1.9077ms" <6> 2023-03-03 09:36:12.585 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:36:12.587 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 2.5326ms" <6> 2023-03-03 09:36:42.584 +00:00 [INF] - "Request starting HTTP/1.1 GET http://edgeHub:9600/metrics - -" <6> 2023-03-03 09:36:42.586 +00:00 [INF] - "Request finished HTTP/1.1 GET http://edgeHub:9600/metrics - - - 200 - text/plain;+version=0.0.4;+charset=utf-8 1.5566ms" <6> 2023-03-03 09:38:57.710 +00:00 [INF] - Entering periodic task to reauthenticate connected clients <6> 2023-03-03 09:38:57.710 +00:00 [INF] - Unable to authenticate client 00137A1000011656 with cached service identity 00137A1000011656 (Found: False). Resyncing service identity... <6> 2023-03-03 09:38:57.710 +00:00 [INF] - Refreshing service identity for 00137A1000011656 <4> 2023-03-03 09:38:57.881 +00:00 [WRN] - Error while refreshing the service identity: 00137A1000011656 OnBehalfOf: carsi4iiotedge1 Microsoft.Azure.Devices.Edge.Hub.Core.DeviceInvalidStateException: Device not in scope: [Unauthorized: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:a0b467104fd84c89a1ec069719a6664b-G:0-TimeStamp:03/03/2023 09:38:57"}]. ---> Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceScopeApiException: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:a0b467104fd84c89a1ec069719a6664b-G:0-TimeStamp:03/03/2023 09:38:57"} at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfInternalAsync(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 191 at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 137 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfWithRetry(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 140 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 at Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeIdentitiesCache.RefreshServiceIdentityInternal(String refreshTarget, String onBehalfOfDevice, Boolean invokeServiceIdentitiesUpdated) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/DeviceScopeIdentitiesCache.cs:line 201 <6> 2023-03-03 09:38:57.882 +00:00 [INF] - Removed node: 00137A1000011656 <6> 2023-03-03 09:38:57.882 +00:00 [INF] - Device scope identities refresh is ready because a refresh was signalled. <6> 2023-03-03 09:38:57.882 +00:00 [INF] - Starting refresh of device scope identities cache <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1 <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/$edgeAgent <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/$edgeHub <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/GenericDecoder <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/MetadataEnricher <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/NkeSensorsDecoder <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/NetvoxSensorsDecoder <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/NKESensorsDecoderPy <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/Kafka2Pi <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/LoRaWanNetworkSrvModule <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/metricbeat <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/filebeat <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Not changed node: carsi4iiotedge1/DefenderIotMicroAgent <6> 2023-03-03 09:38:58.755 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes. <6> 2023-03-03 09:38:58.756 +00:00 [INF] - Unable to authenticate client 70B3D5E75E00CA92 with cached service identity 70B3D5E75E00CA92 (Found: False). Resyncing service identity... <6> 2023-03-03 09:38:58.756 +00:00 [INF] - Refreshing service identity for 70B3D5E75E00CA92 <4> 2023-03-03 09:38:59.281 +00:00 [WRN] - Error while refreshing the service identity: 70B3D5E75E00CA92 OnBehalfOf: carsi4iiotedge1 Microsoft.Azure.Devices.Edge.Hub.Core.DeviceInvalidStateException: Device not in scope: [Unauthorized: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:068258dd7a8c49c88448dfaf64a440dc-G:0-TimeStamp:03/03/2023 09:38:59"}]. ---> Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceScopeApiException: Message: Error getting device scope result from upstream, HttpStatusCode: Unauthorized, Content: {"Message":"ErrorCode:IotHubUnauthorizedAccess;Unauthorized","ExceptionMessage":"Tracking ID:068258dd7a8c49c88448dfaf64a440dc-G:0-TimeStamp:03/03/2023 09:38:59"} at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfInternalAsync(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 191 at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 137 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.NestedDeviceScopeApiClient.GetIdentityOnBehalfOfWithRetry(Uri uri, String deviceId, Option`1 moduleId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/NestedDeviceScopeApiClient.cs:line 140 at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ServiceProxy.GetServiceIdentity(String deviceId, String onBehalfOfDevice) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ServiceProxy.cs:line 101 at Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeIdentitiesCache.RefreshServiceIdentityInternal(String refreshTarget, String onBehalfOfDevice, Boolean invokeServiceIdentitiesUpdated) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/DeviceScopeIdentitiesCache.cs:line 201 <6> 2023-03-03 09:38:59.282 +00:00 [INF] - Removed node: 70B3D5E75E00CA92 <6> 2023-03-03 09:38:59.283 +00:00 [INF] - Skipping cache refresh, waiting 119 seconds until refreshing again. ```
We still have refreshing issues, but I did not see any issues for the gateway, only for the IoT sensors.

By the way, the error in the identityd logs just stops with no reason 2 days ago 😶

identityd logs here ``` Mar 01 08:19:20 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:20Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:19:20 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:20Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:23 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:23Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:23 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:23Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:23 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:23Z [INFO] - <-- GET /identities/identity?api-version=2020-09-01 {"host": "foo", "user-agent": "libcurl-agent/1.0", "accept": "*/*"} Mar 01 08:19:26 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:26Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:26 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:26Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [ERR!] - !!! internal error Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [ERR!] - !!! caused by: master identity key error Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [ERR!] - !!! caused by: error trying to connect: Connection refused (os error 111) Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [ERR!] - !!! caused by: Connection refused (os error 111) Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [INFO] - --> 500 {"content-type": "application/json"} Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:29 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:29Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:19:32 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:32Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:32 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:32Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:35 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:35Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:35 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:35Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [ERR!] - !!! internal error Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [ERR!] - !!! caused by: master identity key error Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [ERR!] - !!! caused by: error trying to connect: Connection refused (os error 111) Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [ERR!] - !!! caused by: Connection refused (os error 111) Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [INFO] - --> 500 {"content-type": "application/json"} Mar 01 08:19:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:38Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:41Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:41Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:44Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:44Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [ERR!] - !!! internal error Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [ERR!] - !!! caused by: master identity key error Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [ERR!] - !!! caused by: error trying to connect: Connection refused (os error 111) Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [ERR!] - !!! caused by: Connection refused (os error 111) Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [INFO] - --> 500 {"content-type": "application/json"} Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:19:47 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:47Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:50Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:50Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:53 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:53Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:53 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:53Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:53 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:53Z [INFO] - <-- GET /identities/identity?api-version=2020-09-01 {"host": "foo", "user-agent": "libcurl-agent/1.0", "accept": "*/*"} Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [ERR!] - !!! internal error Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [ERR!] - !!! caused by: master identity key error Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [ERR!] - !!! caused by: error trying to connect: Connection refused (os error 111) Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [ERR!] - !!! caused by: Connection refused (os error 111) Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [INFO] - --> 500 {"content-type": "application/json"} Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:56Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:19:59 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:59Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:19:59 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:19:59Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:20:02 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:02Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:20:02 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:02Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111) Mar 01 08:20:05 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:05Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 08:20:05 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:05Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 08:20:05 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:05Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:20:05 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:05Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 08:20:05 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:05Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:20:06 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:20:06Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 08:51:07 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:51:07Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 08:51:08 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T08:51:08Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:10:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:10:41Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 09:10:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:10:41Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:11:07 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:11:07Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 09:11:07 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:11:07Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:21:08 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:21:08Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 09:21:08 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:21:08Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:51:09 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:51:09Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 09:51:09 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:51:09Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:59:10 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:59:10Z [INFO] - <-- GET /identities/modules?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 09:59:11 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:59:11Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 09:59:11 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:59:11Z [INFO] - <-- PUT /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 09:59:11 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T09:59:11Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:00:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:00:56Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:00:56 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:00:56Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:01:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:01:42Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:01:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:01:42Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:02:13 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:02:13Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:02:14 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:02:14Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:03:40 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:03:40Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:03:40 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:03:40Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:05:26 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:26Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:05:26 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:26Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:05:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:50Z [INFO] - <-- GET /identities/modules?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:05:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:50Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:05:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:50Z [INFO] - <-- PUT /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:05:50 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:05:50Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:32 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:32Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:07:32 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:32Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:36 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:36Z [INFO] - <-- GET /identities/modules?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - <-- PUT /identities/modules/GenericDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": > Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - <-- PUT /identities/modules/NkeSensorsDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host> Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:37 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:37Z [INFO] - <-- PUT /identities/modules/filebeat?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - <-- PUT /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - <-- PUT /identities/modules/metricbeat?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ide> Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:38 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:38Z [INFO] - <-- PUT /identities/modules/LoRaWanNetworkSrvModule?api-version=2020-09-01&type=aziot {"content-type": "application/json",> Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - <-- PUT /identities/modules/NetvoxSensorsDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "h> Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - <-- PUT /identities/modules/MetadataEnricher?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host"> Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:39 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:39Z [INFO] - <-- PUT /identities/modules/NKESensorsDecoderPy?api-version=2020-09-01&type=aziot {"content-type": "application/json", "ho> Mar 01 10:07:40 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:40Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:07:40 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:40Z [INFO] - <-- PUT /identities/modules/Kafka2Pi?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:07:40 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:07:40Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:41Z [INFO] - <-- GET /identities/modules?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:08:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:41Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:41 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:41Z [INFO] - <-- PUT /identities/modules/GenericDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": > Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - <-- PUT /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - <-- PUT /identities/modules/filebeat?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - <-- PUT /identities/modules/NkeSensorsDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host> Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:42 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:42Z [INFO] - <-- PUT /identities/modules/metricbeat?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ide> Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - <-- PUT /identities/modules/LoRaWanNetworkSrvModule?api-version=2020-09-01&type=aziot {"content-type": "application/json",> Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - <-- PUT /identities/modules/NetvoxSensorsDecoder?api-version=2020-09-01&type=aziot {"content-type": "application/json", "h> Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:43 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:43Z [INFO] - <-- PUT /identities/modules/MetadataEnricher?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host"> Mar 01 10:08:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:44Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:44Z [INFO] - <-- PUT /identities/modules/NKESensorsDecoderPy?api-version=2020-09-01&type=aziot {"content-type": "application/json", "ho> Mar 01 10:08:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:44Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:08:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:44Z [INFO] - <-- PUT /identities/modules/Kafka2Pi?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "ident> Mar 01 10:08:44 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:08:44Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:09:02 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:09:02Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:09:02 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:09:02Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:09:03 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:09:03Z [INFO] - <-- GET /identities/modules/$edgeHub?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} Mar 01 10:09:03 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:09:03Z [INFO] - --> 200 {"content-type": "application/json"} Mar 01 10:09:23 carsi4iiotedge1 aziot-identityd[3884456]: 2023-03-01T10:09:23Z [INFO] - <-- GET /identities/modules/LoRaWanNetworkSrvModule?api-version=2020-09-01&type=aziot {"host": "identityd.sock"} ```

( N.B.: sorry for the logs display, code section is not taken into account. I had logs in a linked file. Identity_service_Logs_and_LoRaWanNetworkSrvModule_Logs.txt )

@nyanzebra
Copy link
Contributor

Based on the error @adcoly it appears that keyd is not actually up and running or there might be some socket isssue for keyd.

I believe you should see the socket for keyd here:
ls /var/run/aziot/
certd.sock identityd.sock keyd.sock tpmd.sock

@jlian
Copy link
Member

jlian commented Mar 9, 2023

@adcoly did it help?

@adcoly
Copy link
Author

adcoly commented Mar 10, 2023 •
edited

(Sorry for the late answer)
I have all the 4 sockets you wrote previously :

~# ls -l /var/run/aziot/
total 0
srw-rw---- 1 aziotcs  aziotcs  0 Mar  1 08:21 certd.sock
srw-rw---- 1 aziotid  aziotid  0 Mar  1 08:04 identityd.sock
srw-rw---- 1 aziotks  aziotks  0 Mar  1 08:20 keyd.sock
srw-rw---- 1 aziottpm aziottpm 0 Feb 23 15:35 tpmd.sock

@nyanzebra
Copy link
Contributor

@adcoly hmm interesting, so you have all the sockets and if you do journalctl -u aziot-keyd you don't see any errors or problems?

@adcoly
Copy link
Author

adcoly commented Mar 14, 2023

I have almost no errors (only two few days ago as shown below)

root@myIotEdge:~# journalctl -u aziot-keyd | grep 500
Mar 01 08:34:33 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-01T08:34:33Z [INFO] - --> 500 {"content-type": "application/json"}
Mar 01 08:42:28 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-01T08:42:28Z [INFO] - --> 500 {"content-type": "application/json"}

(By the way I don't know how often this module has to do requests but has you can see just below, I have almost 2000 lines in only one hour)

aziot-keyd logs for 1h 
root@myIotEdge:~# journalctl -u aziot-keyd --since "1 hour ago"
-- Logs begin at Tue 2023-02-28 17:05:57 UTC, end at Tue 2023-03-14 08:21:55 UTC. --
Mar 14 07:23:29 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T07:23:29Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Mar 14 07:23:29 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T07:23:29Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 14 07:23:29 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T07:23:29Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "386"}
Mar 14 07:23:29 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T07:23:29Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 14 07:23:29 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T07:23:29Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
...
Mar 14 08:18:34 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T08:18:34Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 14 08:18:34 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T08:18:34Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": >
Mar 14 08:18:34 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T08:18:34Z [INFO] - --> 200 {"content-type": "application/json"}
Mar 14 08:18:34 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T08:18:34Z [INFO] - <-- POST /sign?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "954"}
Mar 14 08:18:34 carsi4iiotedge1 aziot-keyd[3888765]: 2023-03-14T08:18:34Z [INFO] - --> 200 {"content-type": "application/json"}
lines 1889-1937/1937 (END)

I also noticed I have less issues in my edgeHub than before. I let the logs for the last 24h here if it can help.

edgeHub logs for the last 24 hours 
root@myIotEdge:~# docker logs edgeHub --since 1h
root@myIotEdge:~# docker logs edgeHub --since 24h
<3> 2023-03-13 12:30:54.041 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-13 14:38:55.037 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-13 20:50:54.042 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateways ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-14 03:13:54.014 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-14 05:51:50.734 +00:00 [WRN] - Error creating cloud connection for client <Sensor Dev EUI>
Microsoft.Azure.Devices.Client.Exceptions.DeviceNotFoundException: error(condition:amqp:not-found,description:{"errorCode":404001,"trackingId":"af9d13718c7c485187b44cd468f8c277-G:0-TimeStamp:03/14/2023 05:51:50-G:0-TimeStamp:03/14/2023 05:51:50","message":"<Sensor Dev EUI>","timestampUtc":"2023-03-14T05:51:50.7081939Z"},info:[com.microsoft:is-filtered:True])
 ---> Microsoft.Azure.Amqp.AmqpException: {"errorCode":404001,"trackingId":"af9d13718c7c485187b44cd468f8c277-G:0-TimeStamp:03/14/2023 05:51:50-G:0-TimeStamp:03/14/2023 05:51:50","message":"<Sensor Dev EUI>","timestampUtc":"2023-03-14T05:51:50.7081939Z"}
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenTelemetrySenderLinkAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.Create(ITokenCredentials tokenCredentials, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout, TimeSpan cloudConnectionHangingTimeout, String productInfo, Option`1 modelId) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 92
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IClientCredentials clientCredentials, Action`2 connectionStatusChangedHandler) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 140
<4> 2023-03-14 05:51:50.737 +00:00 [WRN] - Error validating token for <Sensor Dev EUI>with IoTHub
Microsoft.Azure.Devices.Client.Exceptions.DeviceNotFoundException: error(condition:amqp:not-found,description:{"errorCode":404001,"trackingId":"af9d13718c7c485187b44cd468f8c277-G:0-TimeStamp:03/14/2023 05:51:50-G:0-TimeStamp:03/14/2023 05:51:50","message":"<Sensor Dev EUI>","timestampUtc":"2023-03-14T05:51:50.7081939Z"},info:[com.microsoft:is-filtered:True])
 ---> Microsoft.Azure.Amqp.AmqpException: {"errorCode":404001,"trackingId":"af9d13718c7c485187b44cd468f8c277-G:0-TimeStamp:03/14/2023 05:51:50-G:0-TimeStamp:03/14/2023 05:51:50","message":"<Sensor Dev EUI>","timestampUtc":"2023-03-14T05:51:50.7081939Z"}
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenTelemetrySenderLinkAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.Create(ITokenCredentials tokenCredentials, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout, TimeSpan cloudConnectionHangingTimeout, String productInfo, Option`1 modelId) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 92
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IClientCredentials clientCredentials, Action`2 connectionStatusChangedHandler) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 140

@nyanzebra
Copy link
Contributor

nyanzebra commented Mar 15, 2023 •
edited

@adcoly is your device named <Sensor Dev EUI>? If so, would you mind trying without < and >, as those might not be allowed in the device name.

Also, it appears iothub_hostname and source="dps" under provisioning are not allowed together, so you might need to remove one of those based on what you want to do.

@adcoly
Copy link
Author

adcoly commented Mar 17, 2023 •
edited

<Sensor Dev EUI> & <Gateway ID> are just here to anonymise the real Dev EUI & Gateways (there is no < nor > in the real ones)

I tried this modification and let you know later in the day if there is any news 😃

@jlian
Copy link
Member

jlian commented Mar 21, 2023

@adcoly any updates?

@adcoly
Copy link
Author

adcoly commented Mar 22, 2023

Sorry for the delay,

It seems we still have some issues, but less than before. Here are the edgeHub logs for the last 3 days.

edgeHub logs (since 3 days) 

root@carsi4iiotedge1:~# docker logs edgeHub --since 4320m
<3> 2023-03-19 16:59:54.011 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-19 23:12:56.003 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-20 00:14:34.036 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver942180' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<3> 2023-03-20 01:16:54.032 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 05:19:53.006 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 07:34:54.035 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 09:42:54.002 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 11:49:35.030 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 15:52:54.051 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-20 19:56:13.034 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-20 20:58:45.367 +00:00 [WRN] - Error getting twin for <Gateway ID>
System.OperationCanceledException: The operation was canceled.
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.SendingAmqpLink.EndSendMessage(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendGetTwinRequest(String correlationId) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 186
<4> 2023-03-20 20:58:45.371 +00:00 [WRN] - Unable to send twin update to <Gateway ID> because ModuleMessages link was not found.
<4> 2023-03-20 20:58:45.372 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver1155144' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<3> 2023-03-20 22:01:04.027 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-20 23:55:17.745 +00:00 [WRN] - Updating reported properties failed 70B3D5E75E00CA92
<3> 2023-03-21 02:06:54.042 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-21 14:20:10.033 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-21 15:25:45.138 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver1962833' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<3> 2023-03-21 16:29:49.034 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-21 19:31:21.634 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver2308253' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<3> 2023-03-22 02:43:04.005 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-22 03:49:26.357 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver2454343' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<4> 2023-03-22 03:55:07.643 +00:00 [WRN] - Error creating cloud connection for client <Sensor 1 DevEUI>
Microsoft.Azure.Devices.Client.Exceptions.DeviceNotFoundException: error(condition:amqp:not-found,description:{"errorCode":404001,"trackingId":"4042f856ac454ae4a6ba5c1e95c6d47c-G:0-TimeStamp:03/22/2023 03:55:07-G:0-TimeStamp:03/22/2023 03:55:07","message":"<Sensor 1 DevEUI>","timestampUtc":"2023-03-22T03:55:07.6205504Z"},info:[com.microsoft:is-filtered:True])
 ---> Microsoft.Azure.Amqp.AmqpException: {"errorCode":404001,"trackingId":"4042f856ac454ae4a6ba5c1e95c6d47c-G:0-TimeStamp:03/22/2023 03:55:07-G:0-TimeStamp:03/22/2023 03:55:07","message":"<Sensor 1 DevEUI>","timestampUtc":"2023-03-22T03:55:07.6205504Z"}
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenTelemetrySenderLinkAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.Create(ITokenCredentials tokenCredentials, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout, TimeSpan cloudConnectionHangingTimeout, String productInfo, Option`1 modelId) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 92
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IClientCredentials clientCredentials, Action`2 connectionStatusChangedHandler) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 140
<4> 2023-03-22 03:55:07.650 +00:00 [WRN] - Error validating token for <Sensor 1 DevEUI> with IoTHub
Microsoft.Azure.Devices.Client.Exceptions.DeviceNotFoundException: error(condition:amqp:not-found,description:{"errorCode":404001,"trackingId":"4042f856ac454ae4a6ba5c1e95c6d47c-G:0-TimeStamp:03/22/2023 03:55:07-G:0-TimeStamp:03/22/2023 03:55:07","message":"<Sensor 1 DevEUI>","timestampUtc":"2023-03-22T03:55:07.6205504Z"},info:[com.microsoft:is-filtered:True])
 ---> Microsoft.Azure.Amqp.AmqpException: {"errorCode":404001,"trackingId":"4042f856ac454ae4a6ba5c1e95c6d47c-G:0-TimeStamp:03/22/2023 03:55:07-G:0-TimeStamp:03/22/2023 03:55:07","message":"<Sensor 1 DevEUI>","timestampUtc":"2023-03-22T03:55:07.6205504Z"}
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenSendingAmqpLinkAsync(DeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSession.OpenTelemetrySenderLinkAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.Create(ITokenCredentials tokenCredentials, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout, TimeSpan cloudConnectionHangingTimeout, String productInfo, Option`1 modelId) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 92
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IClientCredentials clientCredentials, Action`2 connectionStatusChangedHandler) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 140
<3> 2023-03-22 04:51:46.013 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<3> 2023-03-22 06:56:54.010 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-22 08:02:35.911 +00:00 [WRN] - Error processing message in TwinReceiving link for <Gateway ID>
Microsoft.Azure.Amqp.AmqpException: 'receiver2454581' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.EdgeReceivingAmqpLink.DisposeMessage(AmqpMessage amqpMessage, Outcome outcome, Boolean settled, Boolean batchable) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/EdgeAmqpLink.cs:line 51
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ReceivingLinkHandler.cs:line 99
<3> 2023-03-22 09:04:55.045 +00:00 [ERR] - Error creating or updating the cloud proxy for client <Gateway ID>
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.<>c.<.cctor>b__27_0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 114
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 160
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceClientWrapper.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/DeviceClientWrapper.cs:line 60
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.OpenAsync() in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 65
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 140
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 123
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ClientTokenCloudConnection.UpdateTokenAsync(ITokenCredentials newTokenCredentials) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs:line 170
<4> 2023-03-22 12:38:45.621 +00:00 [WRN] - Did not receive ack for message 4be769b9-4e2a-4e04-a12a-4eef22dd9e1a from device/module carsi4iiotedge1/MetadataEnricher
<4> 2023-03-22 12:38:45.625 +00:00 [WRN] - Error sending messages to module carsi4iiotedge1/MetadataEnricher
System.TimeoutException: Message completion response not received
   at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 494
   at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.ProcessAsync(ICollection`1 routingMessages, IDeviceProxy dp, CancellationToken token) in /mnt/vss/_work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 167
<4> 2023-03-22 12:38:58.998 +00:00 [WRN] - Received unknown feedback message from carsi4iiotedge1/MetadataEnricher with lock token 4be769b9-4e2a-4e04-a12a-4eef22dd9e1a and status Complete. Abandoning message.

N.B.: <Gateway ID> & <Sensor 1 DevEUI> stand for real gateways id and devEUI

@nyanzebra
Copy link
Contributor

@adcoly would you mind also putting the output of iotedge check after making the recent changes? It appears edgeHub is unable to communicate upstream whatsoever...

@adcoly
Copy link
Author

adcoly commented Mar 29, 2023

Here is a check with verbose option:

iotedge check --verbose 
root@carsi4iiotedge1:~# iotedge check --verbose

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
× aziot-identity-service package is up-to-date - Error
    could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: error trying to connect: tcp connect error: Connection timed out (os error 110)
        caused by: tcp connect error: Connection timed out (os error 110)
        caused by: Connection timed out (os error 110)
‼ host time is close to reference time - Warning
    Could not query NTP server
        caused by: Could not query NTP server
        caused by: could not receive NTP server response: Resource temporarily unavailable (os error 11)
        caused by: Resource temporarily unavailable (os error 11)
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
‼ configuration up-to-date with config.toml - Warning
    /etc/aziot/config.toml was modified after edged's config
    You must run 'iotedge config apply' to update edged's config with the latest config.toml
        caused by: /etc/aziot/config.toml was modified after edged's config
                   You must run 'iotedge config apply' to update edged's config with the latest config.toml
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
    docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers).
    See 'docker run --help'.
        caused by: Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers).
                   See 'docker run --help'.
        caused by: docker returned exit status: 125, stderr = Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers).
                   See 'docker run --help'.
× aziot-edge package is up-to-date - Error
    Error while fetching latest versions of edge components: could not send HTTP request
        caused by: Error while fetching latest versions of edge components: could not send HTTP request
        caused by: error trying to connect: tcp connect error: Connection timed out (os error 110)
        caused by: tcp connect error: Connection timed out (os error 110)
        caused by: Connection timed out (os error 110)
× container time is close to host time - Error
    Could not query local time inside container
        caused by: Could not query local time inside container
        caused by: docker returned exit status: 125, stderr = Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
                   See 'docker run --help'.
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
                   Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
                   You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Container engine is not configured to rotate module logs which may cause it run out of disk space.
                   Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
                   You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ Agent image is valid and can be pulled from upstream - Warning
    skipping because of previous failures
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
‼ container on the default network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream MQTT port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream MQTT port - Warning
    skipping because of previous failures
18 check(s) succeeded.
9 check(s) raised warnings.
4 check(s) raised errors.
7 check(s) were skipped due to errors from other checks.

@nyanzebra
Copy link
Contributor

@adcoly would you mind running iotedge config apply? It will regenerate some of the configuration files to match any recent changes you have done. Based on the check it appears that all network required tasks are failing due to timeout.

@adcoly
Copy link
Author

adcoly commented Apr 3, 2023 •
edited

Here is the check after an apply :

iotedge check --verbose 
root@carsi4iiotedge1:~# iotedge check --verbose

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK

× aziot-identity-service package is up-to-date - Error
    could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: could not query https://aka.ms/latest-aziot-identity-service for latest available version
        caused by: error trying to connect: tcp connect error: Connection timed out (os error 110)
        caused by: tcp connect error: Connection timed out (os error 110)
        caused by: Connection timed out (os error 110)
‼ host time is close to reference time - Warning
    Could not query NTP server
        caused by: Could not query NTP server
        caused by: could not receive NTP server response: Resource temporarily unavailable (os error 11)
        caused by: Resource temporarily unavailable (os error 11)
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
    docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded.
    See 'docker run --help'.
        caused by: Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded.
                   See 'docker run --help'.
        caused by: docker returned exit status: 125, stderr = Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": context deadline exceeded.
                   See 'docker run --help'.
× aziot-edge package is up-to-date - Error
    Error while fetching latest versions of edge components: could not send HTTP request
        caused by: Error while fetching latest versions of edge components: could not send HTTP request
        caused by: error trying to connect: tcp connect error: Connection timed out (os error 110)
        caused by: tcp connect error: Connection timed out (os error 110)
        caused by: Connection timed out (os error 110)
× container time is close to host time - Error
    Could not query local time inside container
        caused by: Could not query local time inside container
        caused by: docker returned exit status: 125, stderr = Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.9' locally
                   docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
                   See 'docker run --help'.
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
                   Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
                   You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Container engine is not configured to rotate module logs which may cause it run out of disk space.
                   Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
                   You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ Agent image is valid and can be pulled from upstream - Warning
    skipping because of previous failures
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
‼ container on the default network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream MQTT port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream MQTT port - Warning
    skipping because of previous failures
19 check(s) succeeded.
8 check(s) raised warnings.
4 check(s) raised errors.
7 check(s) were skipped due to errors from other checks.

I'll check if there is less connectivity issues in the edgeHub in the following hours.

@nyanzebra
Copy link
Contributor

@adcoly still looks like connectivity from modules to outside is broken... if you docker exec into one of the nodes are you able to ping or make any outside network calls?

Also, would you mind running this on the device? docker run -it --rm --name ubuntu ubuntu /bin/bash -c "apt-get update"

@bishal41
Copy link

@adcoly any update on this? Were you able to test the suggestion provided above?

@bishal41
Copy link

@adcoly let me know if I can go ahead and close this issue?

@nyanzebra
Copy link
Contributor

@bishal41 let's just close for now, if @adcoly wants to re-open that is always an option :)

@konichi3
Copy link

konichi3 commented May 4, 2023

Closing the issue as there is no response. @adcoly feel free to re-open or create a new item if necessary

@konichi3 konichi3 closed this as completed May 4, 2023
@Metal-Mighty Metal-Mighty mentioned this issue Mar 21, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nyanzebra nyanzebra

Labels
customer-reported iotedge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jlian @nyanzebra @konichi3 @adcoly @bishal41