az extension add -n azure-cli-ml results in 500

[lostmygithubaccount]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
Running az extension add -n azure-cli-ml frequently results in 500 error.

To Reproduce
Run the command in a GitHub action fairly frequently :)

Expected behavior
azure ml cli extension is installed without hitting 500

Environment summary
github action on standard ubuntu agents

Additional context

[yonzhan]

azure-cli-ml

[yungezz]

hi @fengzhou-msft could you pls help to check availability of the storage url?

[lostmygithubaccount]

@fengzhou-msft @yungezz any updates here? this is increasingly becoming a point of failure in our automated testing for Azure Machine Learning

[fengzhou-msft]

We have also seen increasing failures for this aka.ms link. We will work with the Redirection Service team to investigate it.

As a workaround, you can get the package URL from here and install with
az extension add -s https://azurecliext.blob.core.windows.net/release/azure_cli_ml-1.15.0-py3-none-any.whl -y to bypass the index link.

[lostmygithubaccount]

can running az extension add -n azure-cli-ml be changed to that link instead?

[fengzhou-msft]

can running az extension add -n azure-cli-ml be changed to that link instead?

We can't change it directly. The link is an index of all extensions, CLI looks for the download link for azure-cli-ml extension or any other extensions in it.

We will see how we can improve it once we identify the root cause of the issue.

[lostmygithubaccount]

@fengzhou-msft any updates?

[fengzhou-msft]

We encountered other connection failures quite frequently recently while downloading different extensions using storage account links directly in Azure Deveops. It looks more like network issues with the CI environment.

[fengzhou-msft]

We contacted CSS team and got the following response:

As per the issue of random 500 errors for your storage account, our product group has helped to mitigate the issue on the backend.

Regarding the root cause of the issue, one of the backend nodes got an unusual spike of incoming requests, causing higher latencies on the node. As for now, on the Azure Storage side, we are working towards making improvements to detect these spikes sooner and remove such problematic nodes out of service sooner.

cc: @jiasli

[fengzhou-msft]

We've added retry for fetching the extension index file. This issue should be mitigated.