[BUG] GenericResources.ExistAsync is unreliable

[krizex]

Library name and version

Azure.ResourceManager 1.10.1

Describe the bug

GenericResources.ExistAsync is unreliable when the target Azure service for the resource is upgrading.
For example, to check the existance of

subscriptions/<xxxxxx>/resourceGroups/<xxxxx>/providers/Microsoft.Network/networkSecurityGroups/myNsg

by using the following code

        await client.GetGenericResources().ExistsAsync(identifier).ConfigureAwait(false);

It sometimes failed with the following error. (Starts from March 01, 2024 I got the issue in eastus)

Azure.RequestFailedException: No registered resource provider found for location 'eastus' and API version '2023-11-01' for type 'networkSecurityGroups'. The supported api-versions are '2014-12-01-preview, 2015-05-01-preview, 2015-06-15, 2016-03-30, 2016-06-01, 2016-07-01, 2016-08-01, 2016-09-01, 2016-10-01, 2016-11-01, 2016-12-01, 2017-03-01, 2017-04-01, 2017-06-01, 2017-08-01, 2017-09-01, 2017-10-01, 2017-11-01, 2018-01-01, 2018-02-01, 2018-03-01, 2018-04-01, 2018-05-01, 2018-06-01, 2018-07-01, 2018-08-01, 2018-10-01, 2018-11-01, 2018-12-01, 2019-02-01, 2019-04-01, 2019-06-01, 2019-07-01, 2019-08-01, 2019-09-01, 2019-11-01, 2019-12-01, 2020-01-01, 2020-03-01, 2020-04-01, 2020-05-01, 2020-06-01, 2020-07-01, 2020-08-01, 2020-11-01, 2021-01-01, 2021-02-01, 2021-03-01, 2021-04-01, 2021-05-01, 2021-06-01, 2021-08-01, 2021-12-01, 2022-01-01, 2022-05-01, 2022-07-01, 2022-09-01, 2022-11-01, 2023-02-01, 2023-04-01, 2023-05-01, 2023-06-01, 2023-09-01'. The supported locations are 'westus, eastus, northeurope, westeurope, eastasia, southeastasia, northcentralus, southcentralus, centralus, eastus2, japaneast, japanwest, brazilsouth, australiaeast, australiasoutheast, centralindia, southindia, westindia, canadacentral, canadaeast, westcentralus, westus2, ukwest, uksouth, koreacentral, koreasouth, francecentral, australiacentral, southafricanorth, uaenorth, switzerlandnorth, germanywestcentral, norwayeast, westus3, jioindiawest, swedencentral, qatarcentral, polandcentral, italynorth, israelcentral'.
Status: 400 (Bad Request)
ErrorCode: NoRegisteredProviderFound

Content:
{"error":{"code":"NoRegisteredProviderFound","message":"No registered resource provider found for location 'eastus' and API version '2023-11-01' for type 'networkSecurityGroups'. The supported api-versions are '2014-12-01-preview, 2015-05-01-preview, 2015-06-15, 2016-03-30, 2016-06-01, 2016-07-01, 2016-08-01, 2016-09-01, 2016-10-01, 2016-11-01, 2016-12-01, 2017-03-01, 2017-04-01, 2017-06-01, 2017-08-01, 2017-09-01, 2017-10-01, 2017-11-01, 2018-01-01, 2018-02-01, 2018-03-01, 2018-04-01, 2018-05-01, 2018-06-01, 2018-07-01, 2018-08-01, 2018-10-01, 2018-11-01, 2018-12-01, 2019-02-01, 2019-04-01, 2019-06-01, 2019-07-01, 2019-08-01, 2019-09-01, 2019-11-01, 2019-12-01, 2020-01-01, 2020-03-01, 2020-04-01, 2020-05-01, 2020-06-01, 2020-07-01, 2020-08-01, 2020-11-01, 2021-01-01, 2021-02-01, 2021-03-01, 2021-04-01, 2021-05-01, 2021-06-01, 2021-08-01, 2021-12-01, 2022-01-01, 2022-05-01, 2022-07-01, 2022-09-01, 2022-11-01, 2023-02-01, 2023-04-01, 2023-05-01, 2023-06-01, 2023-09-01'. The supported locations are 'westus, eastus, northeurope, westeurope, eastasia, southeastasia, northcentralus, southcentralus, centralus, eastus2, japaneast, japanwest, brazilsouth, australiaeast, australiasoutheast, centralindia, southindia, westindia, canadacentral, canadaeast, westcentralus, westus2, ukwest, uksouth, koreacentral, koreasouth, francecentral, australiacentral, southafricanorth, uaenorth, switzerlandnorth, germanywestcentral, norwayeast, westus3, jioindiawest, swedencentral, qatarcentral, polandcentral, italynorth, israelcentral'."}}

Headers:
Cache-Control: no-cache
Pragma: no-cache
x-ms-failure-cause: REDACTED
x-ms-request-id: 0bb0ddde-785b-4612-8dc9-d204313a9942
x-ms-correlation-request-id: REDACTED
x-ms-routing-request-id: REDACTED
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
X-Cache: REDACTED
X-MSEdge-Ref: REDACTED
Date: Thu, 07 Mar 2024 02:30:39 GMT
Content-Length: 1540
Content-Type: application/json; charset=utf-8
Expires: -1

From my understanding of the code, I think that's because Azure.ResourceManager SDK uses resource provider to detect the latest api-version of the specific resource type, and uses that api-version to query the resource servier. However, the specific api-version may not be available for the service as the service is upgrading. (I think it is rollong upgrade and should take some time to complete for all services)

Some snapshot of my tenant and requests:
Resource provider shows the api-version

Sometimes success

Sometimes failed

Expected behavior

Always return the correct result.

Actual behavior

Not reliable

Reproduction Steps

Check out the description

Environment

No response

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[ArthurMa1978]

Hi @krizex, as you said this happens when service is upgrading, so this issue is related to the service side and not the SDK side, that we are not able to help you with it. Therefore, we are closing this issue for now. If you need more assistance, please open an Azure support request. We apologize for any inconvenience this may cause you.

We value your feedback and we want to make sure that your problem is solved. If you think that we have misunderstood your issue or closed it incorrectly, please feel free to comment on this thread and reopen the issue. We will be happy to assist you further. Thank you for your understanding and cooperation.

[krizex]

@ArthurMa1978 I have opened an Azure support ticket and Microsoft replied me it is caused by "the absence of provider Microsoft/Network containing the new API Version (2023-11-01) in a few Regions. And the backend team have to fix it."
Actually the bug is fixed 2 days later, but I am wondering could the SDK team consider some kind of toleration to make the client.GetGenericResources().ExistsAsync(identifier) still works if some Azure service fails to upgrade completely? E.g. use the second latest api-version instead of the latest one to check the existence of the resource?