ai-exp6 - knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run az login for this service principal.

[krisbock]

Task 2: Run the Build Pipeline fails in 'Get or create AML Compute Target' step with:

2022-01-06T05:53:56.435590] Logging experiment finalizing status in history service.
Starting the daemon thread to refresh tokens in background for process with pid = 3694
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.1008906364440918 seconds
Traceback (most recent call last):
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azureml/core/authentication.py", line 2283, in _get_arm_token_with_refresh
access_token = profile_object.get_raw_token(resource=resource)[0][1]
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 383, in get_raw_token
credential = self._create_credential(account, tenant)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 597, in _create_credential
return identity.get_service_principal_credential(username_or_sp_id)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 236, in get_service_principal_credential
entry = self._service_principal_store.load_entry(client_id, self.tenant_id)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 320, in load_entry
.format(sp_id))
knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run az login for this service principal.

Error from AML driver log:
n create_aml_cluster.py
Azure ML SDK version: 1.37.0
Argument 1: gpucluster
Argument 2: /home/vsts/work/1/s
creating AzureCliAuthentication...
done creating AzureCliAuthentication!
get workspace...

[2022-01-06T05:53:56.425879] The experiment failed. Finalizing run...
[2022-01-06T05:53:56.425904] Start FinalizingInRunHistory
[2022-01-06T05:53:56.435590] Logging experiment finalizing status in history service.
Starting the daemon thread to refresh tokens in background for process with pid = 3694
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.1008906364440918 seconds
Traceback (most recent call last):
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azureml/core/authentication.py", line 2283, in _get_arm_token_with_refresh
access_token = profile_object.get_raw_token(resource=resource)[0][1]
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 383, in get_raw_token
credential = self._create_credential(account, tenant)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 597, in _create_credential
return identity.get_service_principal_credential(username_or_sp_id)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 236, in get_service_principal_credential
entry = self._service_principal_store.load_entry(client_id, self.tenant_id)
File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 320, in load_entry
.format(sp_id))
knack.util.CLIError: Could not retrieve credential from local cache for service principal XXXXXXXXXXXXX. Run az login for this service principal.

Looks like the error is caused by breaking changing in azcli with tokens: Azure/cli#56

[krisbock]

Looks like a number of PRs have been issued for the source repo which needs the CLI version updated: solliancenet/mcw-mlops-starter#3

i.e. updating environment_setup/install_requirements.sh to
python --version
pip install azure-cli==2.32.0
pip install azureml-sdk==1.37.0
pip install azureml-dataprep[pandas]