-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
I'm experiencing an error while trying to deploy the sample ML Model on ACI/AKS. This is the log for the Model deployment on AKS
2020-03-13T09:59:39.3192490Z ##[section]Starting: Azure ML Model Deploy
2020-03-13T09:59:39.3202122Z ==============================================================================
2020-03-13T09:59:39.3202907Z Task : AzureML Model Deploy
2020-03-13T09:59:39.3203403Z Description : AzureML Deploy Model
2020-03-13T09:59:39.3203836Z Version : 0.8.20
2020-03-13T09:59:39.3204272Z Author : ms-air-aiagility
2020-03-13T09:59:39.3205010Z Help : https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.private-vss-services-azureml&ssr=false#overview
2020-03-13T09:59:39.3205673Z ==============================================================================
2020-03-13T09:59:39.7985527Z Current Working Directory is : /__w/1/s
2020-03-13T09:59:39.7986831Z Desired Working Directory is : /__w/1/s/diabetes_regression/scoring
2020-03-13T09:59:39.7987391Z Setting Working Directory ......
2020-03-13T09:59:39.7993718Z /__w/1/s/diabetes_regression/scoring /__w/1/s
2020-03-13T09:59:39.7994421Z Current Working Directory : /__w/1/s/diabetes_regression/scoring
2020-03-13T09:59:39.8059477Z ##[warning]Resource file haven't set, can't find loc string for key: AmlCliNotAvailable
2020-03-13T09:59:39.8072218Z [command]/usr/local/envs/mlopspython_ci/bin/az ml -h
2020-03-13T09:59:40.4705260Z ERROR: az: 'ml' is not in the 'az' command group. See 'az --help'. If the command is from an extension, please make sure the corresponding extension is installed. To learn more about extensions, please visit https://docs.microsoft.com/en-us/cli/azure/azure-cli-extensions-overview
2020-03-13T09:59:40.4706314Z No AzureML CLI installed.
2020-03-13T09:59:40.4714830Z [command]/usr/local/envs/mlopspython_ci/bin/az extension add -n azure-cli-ml
2020-03-13T09:59:54.5377696Z [command]/usr/local/envs/mlopspython_ci/bin/az ml -h
2020-03-13T09:59:56.1414052Z
2020-03-13T09:59:56.1414981Z Group
2020-03-13T09:59:56.1415305Z az ml : Azureml._cli commands.
2020-03-13T09:59:56.1415550Z
2020-03-13T09:59:56.1415820Z Subgroups:
2020-03-13T09:59:56.1416127Z computetarget : Computetarget subgroup commands.
2020-03-13T09:59:56.1416731Z dataset : Commands for managing datasets in Azure Machine Learning Workspace.
2020-03-13T09:59:56.1418039Z datastore : Commands for managing and using datastores with the Azure ML Workspace.
2020-03-13T09:59:56.1418669Z endpoint : Manage machine learning endpoints.
2020-03-13T09:59:56.1419359Z environment : Commands to manage environments.
2020-03-13T09:59:56.1419871Z experiment : Commands to manage experiments.
2020-03-13T09:59:56.1420389Z folder : Folder subgroup commands.
2020-03-13T09:59:56.1421086Z model : Manage machine learning models.
2020-03-13T09:59:56.1421609Z pipeline : Pipeline subgroup commands.
2020-03-13T09:59:56.1422167Z run : Commands for submitting, updating, and monitoring runs.
2020-03-13T09:59:56.1422732Z service : Manage operationalized services.
2020-03-13T09:59:56.1423543Z workspace : Workspace subgroup commands.
2020-03-13T09:59:56.1423917Z
2020-03-13T09:59:56.1424353Z Please let us know how we are doing: https://aka.ms/clihats
2020-03-13T09:59:56.1439294Z [command]/usr/local/envs/mlopspython_ci/bin/az login --service-principal -u *** -p *** --tenant ***
2020-03-13T09:59:58.0916080Z [
2020-03-13T09:59:58.0916581Z {
2020-03-13T09:59:58.0916802Z "cloudName": "AzureCloud",
2020-03-13T09:59:58.0917705Z "id": "f88195e4-48f1-4882-9a6f-de3738ed55ac",
2020-03-13T09:59:58.0918229Z "isDefault": true,
2020-03-13T09:59:58.0918659Z "name": "Enginuity CSP",
2020-03-13T09:59:58.0918966Z "state": "Enabled",
2020-03-13T09:59:58.0919872Z "tenantId": "",
2020-03-13T09:59:58.0920212Z "user": {
2020-03-13T09:59:58.0920844Z "name": "",
2020-03-13T09:59:58.0921156Z "type": "servicePrincipal"
2020-03-13T09:59:58.0921434Z }
2020-03-13T09:59:58.0921660Z }
2020-03-13T09:59:58.0921887Z ]
2020-03-13T09:59:58.0926475Z ##[warning]Resource file haven't set, can't find loc string for key: LoginFailed
2020-03-13T09:59:58.0976163Z [command]/usr/local/envs/mlopspython_ci/bin/az account set --subscription f88195e4-48f1-4882-9a6f-de3738ed55ac
2020-03-13T09:59:59.7655554Z ##[warning]Resource file haven't set, can't find loc string for key: ErrorInSettingUpSubscription
2020-03-13T09:59:59.7659066Z From service connection: resource group = FAenginuity workspace = mlops-AML-WS
2020-03-13T09:59:59.7678276Z Cannot find result.json for the model profile result.
2020-03-13T09:59:59.7681149Z Command: az ml model deploy --ct aks-compute -n aks-deployment --model diabetes_regression_model.pkl:1 --ic /__w/1/s/diabetes_regression/scoring/inference_config.yml --dc /__w/1/s/diabetes_regression/scoring/deployment_config_aks.yml -g FAenginuity -w mlops-AML-WS --overwrite
2020-03-13T09:59:59.7711017Z [command]/usr/local/envs/mlopspython_ci/bin/az ml model deploy --ct aks-compute -n aks-deployment --model diabetes_regression_model.pkl:1 --ic /__w/1/s/diabetes_regression/scoring/inference_config.yml --dc /__w/1/s/diabetes_regression/scoring/deployment_config_aks.yml -g FAenginuity -w mlops-AML-WS --overwrite
2020-03-13T10:00:01.3760399Z ERROR: The command failed with an unexpected error. Here is the traceback:
2020-03-13T10:00:01.3765502Z
2020-03-13T10:00:01.3767886Z ERROR: cannot import name 'COMPUTE_TYPE_KEY' from 'azureml._model_management._constants' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_constants.py)
2020-03-13T10:00:01.3768528Z Traceback (most recent call last):
2020-03-13T10:00:01.3769138Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/cli.py", line 206, in invoke
2020-03-13T10:00:01.3769574Z cmd_result = self.invocation.execute(args)
2020-03-13T10:00:01.3770221Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/init.py", line 528, in execute
2020-03-13T10:00:01.3770684Z self.commands_loader.load_arguments(command)
2020-03-13T10:00:01.3771330Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 299, in load_arguments
2020-03-13T10:00:01.3771865Z self.command_table[command].load_arguments() # this loads the arguments via reflection
2020-03-13T10:00:01.3772571Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/init.py", line 291, in load_arguments
2020-03-13T10:00:01.3773038Z super(AzCliCommand, self).load_arguments()
2020-03-13T10:00:01.3773650Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/commands.py", line 97, in load_arguments
2020-03-13T10:00:01.3774076Z cmd_args = self.arguments_loader()
2020-03-13T10:00:01.3774718Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 496, in default_arguments_loader
2020-03-13T10:00:01.3775429Z op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
2020-03-13T10:00:01.3776389Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 536, in get_op_handler
2020-03-13T10:00:01.3776832Z op = import_module(mod_to_import)
2020-03-13T10:00:01.3777235Z File "/usr/local/envs/mlopspython_ci/lib/python3.7/importlib/init.py", line 127, in import_module
2020-03-13T10:00:01.3777694Z return _bootstrap._gcd_import(name[level:], package, level)
2020-03-13T10:00:01.3778098Z File "", line 1006, in _gcd_import
2020-03-13T10:00:01.3778476Z File "", line 983, in _find_and_load
2020-03-13T10:00:01.3778888Z File "", line 967, in _find_and_load_unlocked
2020-03-13T10:00:01.3779301Z File "", line 677, in _load_unlocked
2020-03-13T10:00:01.3779700Z File "", line 728, in exec_module
2020-03-13T10:00:01.3780118Z File "", line 219, in _call_with_frames_removed
2020-03-13T10:00:01.3780773Z File "/home/vsts_azpcontainer/.azure/cliextensions/azure-cli-ml/azext_ml/model.py", line 8, in
2020-03-13T10:00:01.3781368Z from ._util import cli_context, collect_input_models, file_stream_to_object,
2020-03-13T10:00:01.3782026Z File "/home/vsts_azpcontainer/.azure/cliextensions/azure-cli-ml/azext_ml/_util.py", line 25, in
2020-03-13T10:00:01.3782499Z from azureml._model_management._constants import ACI_WEBSERVICE_TYPE,
2020-03-13T10:00:01.3783330Z ImportError: cannot import name 'COMPUTE_TYPE_KEY' from 'azureml._model_management._constants' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_constants.py)
2020-03-13T10:00:01.3841278Z WARNING:
2020-03-13T10:00:01.3842400Z To open an issue, please run: 'az feedback'
2020-03-13T10:00:01.3842892Z Output:
2020-03-13T10:00:01.3849126Z ##[error]Error Code: [1]
2020-03-13T10:00:01.3851776Z ##[error]Error: Failed to deploy model(s)
2020-03-13T10:00:01.3893581Z { code: 1,
2020-03-13T10:00:01.3896886Z error: undefined,
2020-03-13T10:00:01.3897802Z stdout: '',
2020-03-13T10:00:01.3954639Z stderr: 'ERROR: The command failed with an unexpected error. Here is the traceback:\n\nERROR: cannot import name 'COMPUTE_TYPE_KEY' from 'azureml._model_management._constants' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_constants.py)\nTraceback (most recent call last):\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/cli.py", line 206, in invoke\n cmd_result = self.invocation.execute(args)\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/init.py", line 528, in execute\n self.commands_loader.load_arguments(command)\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 299, in load_arguments\n self.command_table[command].load_arguments() # this loads the arguments via reflection\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/init.py", line 291, in load_arguments\n super(AzCliCommand, self).load_arguments()\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/commands.py", line 97, in load_arguments\n cmd_args = self.arguments_loader()\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 496, in default_arguments_loader\n op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/init.py", line 536, in get_op_handler\n op = import_module(mod_to_import)\n File "/usr/local/envs/mlopspython_ci/lib/python3.7/importlib/init.py", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "", line 1006, in _gcd_import\n File "", line 983, in _find_and_load\n File "", line 967, in _find_and_load_unlocked\n File "", line 677, in _load_unlocked\n File "", line 728, in exec_module\n File "", line 219, in _call_with_frames_removed\n File "/home/vsts_azpcontainer/.azure/cliextensions/azure-cli-ml/azext_ml/model.py", line 8, in \n from ._util import cli_context, collect_input_models, file_stream_to_object, \\n File "/home/vsts_azpcontainer/.azure/cliextensions/azure-cli-ml/azext_ml/_util.py", line 25, in \n from azureml._model_management._constants import ACI_WEBSERVICE_TYPE, \\nImportError: cannot import name 'COMPUTE_TYPE_KEY' from 'azureml._model_management._constants' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_constants.py)\nWARNING: \nTo open an issue, please run: 'az feedback'\n' }
2020-03-13T10:00:01.3980970Z Reverting to original directory.....
2020-03-13T10:00:01.3981549Z /__w/1/s
2020-03-13T10:00:01.4128216Z ##[section]Finishing: Azure ML Model Deploy