Mismatch between message routes in the IoT hub and routes shown in the device update instance details

[0xarmingithub]

Hi. I have followed the official document to create resources and link device update to IoT hub. I added message routes according to the table. However, When I click on the instance, I can see an error about routes. If you compare list of routes in the instance with the actual routes in the IoT hub, you will find a missing route (DEVICE LIFE CYCLE ROUTE) in the instance details page. However, that route is already in the IoT hub's message routes table. And I suspect the instability of device update, that we experience now, is because of this issue. However, we cannot figure out how we can solve it. Furthermore, when I want to re-link IoT hub, It throws not informative error in UI (log is attached). I'm also attaching all the relevant screenshots for your reference.

{
    "authorization": {
        "action": "Microsoft.DeviceUpdate/accounts/instances/write",
        "scope": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.DeviceUpdate/accounts/XXX/instances/XXX"
    },
    "caller": "XXX",
    "channels": "Operation",
    "claims": {
        "aud": "https://management.core.windows.net/",
        "iss": "XXX",
        "iat": "1667481476",
        "nbf": "1667481476",
        "exp": "1667486334",
        "http://schemas.microsoft.com/claims/authnclassreference": "1",
        "aio": "XXX",
        "http://schemas.microsoft.com/claims/authnmethodsreferences": "pwd,rsa,mfa",
        "appid": "XXX",
        "appidacr": "2",
        "http://schemas.microsoft.com/2012/01/devicecontext/claims/identifier": "XXX",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "XXX",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "XXX",
        "groups": "XXX",
        "ipaddr": "XXX",
        "name": "XXX",
        "http://schemas.microsoft.com/identity/claims/objectidentifier": "XXX",
        "onprem_sid": "XXX",
        "puid": "XXX",
        "rh": "XXX",
        "http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "XXX",
        "http://schemas.microsoft.com/identity/claims/tenantid": "XXX",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "XXX",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "XXX",
        "uti": "XXX",
        "ver": "1.0",
        "xms_tcdt": "XXX"
    },
    "correlationId": "XXX",
    "description": "",
    "eventDataId": "XXX",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2022-11-03T13:32:51.5063992Z",
    "id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.DeviceUpdate/accounts/XXX/instances/XXX/events/XXX/ticks/XXX",
    "level": "Error",
    "operationId": "XXX",
    "operationName": {
        "value": "Microsoft.DeviceUpdate/accounts/instances/write",
        "localizedValue": "Create/Update Instance"
    },
    "resourceGroupName": "RG-XXXDev-ConnectedService-Backend",
    "resourceProviderName": {
        "value": "Microsoft.DeviceUpdate",
        "localizedValue": "Microsoft.DeviceUpdate"
    },
    "resourceType": {
        "value": "Microsoft.DeviceUpdate/accounts/instances",
        "localizedValue": "Microsoft.DeviceUpdate/accounts/instances"
    },
    "resourceId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.DeviceUpdate/accounts/XXX/instances/XXX",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "BadRequest",
        "localizedValue": "Bad Request (HTTP Status Code: 400)"
    },
    "submissionTimestamp": "2022-11-03T13:33:56.1455412Z",
    "subscriptionId": "XXX",
    "tenantId": "XXX",
    "properties": {
        "statusCode": "BadRequest",
        "serviceRequestId": null,
        "statusMessage": "{\"error\":{\"code\":\"InstanceValidationFailed\",\"message\":\"ValidationException\",\"details\":[{\"code\":\"NotEmptyValidator\",\"message\":\"'Connection String' must not be empty.\"}]}}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.DeviceUpdate/accounts/XXX/instances/XXX",
        "message": "Microsoft.DeviceUpdate/accounts/instances/write",
        "hierarchy": "XXX/XXX"
    },
    "relatedEvents": []
}

[eshashah-msft]

@arminghasemazar all configurations for the messaging routes and endpoints should be automatically during resource creation. The missing message route in the instance will be fixed but should not cause any issues. We are taking a look at the error you saw.
Can you provide some more details on the instability that you are seeing?

[vimeht]

Also, can you confirm what SKU of IoT Hub you are using. Certain IoT Hub SKUs have a limit on the number of message routes that can be configured which will prevent Device Update from configuring the right set of routes.

[0xarmingithub]

@eshashah-msft I think there has been some changes in Azure back-end and I don't see that error anymore and seems like the link is there. However, I'm facing instability in importing update. I've been trying to import update but they all stuck in "running" status for a while and then failed with message as follows. Please let me know if I can share the Trace ID publicly.

Below is the apt update and its corresponding import manifest.

{
    "name": "ubuntu",
    "version": "1.0.0",
    "packages": [
	{
            "name" : "libiw30",
            "version" : "30~pre9-13ubuntu1"
        },
	{
            "name" : "wireless-tools",
            "version" : "30~pre9-13ubuntu1"
        }
    ]
}

{
  "compatibility": [
    {
      "manufacturer": "XXX",
      "model": "ubuntu"
    }
  ],
  "createdDateTime": "2022-11-08T13:18:52Z",
  "description": "Install two apt packages",
  "files": [
    {
      "filename": "XXX-ubuntu-v1.0.0.updatemanifest.json",
      "hashes": {
        "sha256": "uRKXfTHeBaUIeqPQdbglpcWC5Z5o0GTf5vdtcZNnUKI="
      },
      "sizeInBytes": 256
    }
  ],
  "instructions": {
    "steps": [
      {
        "files": [
          "XXX-ubuntu-v1.0.0.updatemanifest.json"
        ],
        "handler": "microsoft/apt:1",
        "handlerProperties": {
          "installedCriteria": "1.0"
        },
        "type": "inline"
      }
    ]
  },
  "manifestVersion": "5.0",
  "updateId": {
    "name": "ubuntu",
    "provider": "XXX",
    "version": "1.0.0"
  }
}

@vimeht we are using SKU S1 with 1 unit. But we are way below the daily messages quota and each message size is below 4 KB.

[iryanzhang]

Hi @arminghasemazar

Thanks for reporting the import unstable issue.
The underlying service side issue was addressed and could you please confirm the issue is not persisting?

[0xarmingithub]

I can confirm that the service is now fully working. Thanks for the update and great work. We are going to use Azure Device Update service extensively in our organization. Let's see how reliably it would help us in maintaining a large fleet of devices we have. :)