online deployment failed - cannot import name 'json' from 'itsdangerous'

[roalexan]

I'm consistently getting an error when deploying my online endpoint. I can reproduce it using both the cli and arm templates. For example: az ml online-deployment create -f 41_deploy_pretrained.yml -e ${ENDPOINT_NAME} --set environment=${environment} --all-traffic. The error is shown in the deployment logs of the deployed endpoint in the aml workspace:

..
File "/var/azureml-server/aml_framework.py", line 9, in
from synchronous.framework import *
File "/var/azureml-server/synchronous/framework.py", line 3, in
from flask import Flask, request, g, Request, Response, Blueprint
File "/azureml-envs/azureml_7efde0279491eb9584fda4142caa94af/lib/python3.7/site-packages/flask/init.py", line 21, in
from .app import Flask, Request, Response
File "/azureml-envs/azureml_7efde0279491eb9584fda4142caa94af/lib/python3.7/site-packages/flask/app.py", line 26, in
from . import cli, json
File "/azureml-envs/azureml_7efde0279491eb9584fda4142caa94af/lib/python3.7/site-packages/flask/json/init.py", line 21, in
from itsdangerous import json as _json
ImportError: cannot import name 'json' from 'itsdangerous' (/azureml-envs/azureml_7efde0279491eb9584fda4142caa94af/lib/python3.7/site-packages/itsdangerous/init.py)
Worker exiting (pid: 37)
Shutting down: Master
Reason: Worker failed to boot.
2022-02-19T02:55:15,265990495+00:00 - gunicorn/finish 3 0
2022-02-19T02:55:15,268212063+00:00 - Exit code 3 is not normal. Killing image.

I'm reasonable sure it's related to issues such as the one mentioned here . From what I read, the resolution is "Either update Flask to 1.1.4, or fix itsdangerous version to 2.0.1"

I'm not sure how to file this bug with aml, but believe this is something AML needed to fix ASAP

FWIW, below are what the environment and conda yamls looks like, but I don't think this part matters:

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: aml_energy_km_env_non_nlp
version: 1
image: mcr.microsoft.com/azureml/openmpi3.1.2-cuda10.2-cudnn8-ubuntu18.04:20210507.v1
conda_file: conda_geolocation.yml


channels:
  - conda-forge
dependencies:
  - python==3.7.7
  - pip>=19.0.3
  - pip:
      - 'azureml-sdk[notebooks,contrib]==1.32.0'
      - azureml-defaults==1.32.0
      - spacy==3.0.4
      - joblib==1.1.0

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 03c0bed2-1b0d-41e2-bf86-5a8c0956469e
• Version Independent ID: 0f9fc0b8-ea87-ceb4-bdac-fc079d0463e9
• Content: az ml online-deployment
• Content Source: latest/docs-ref-autogen/ml/online-deployment.yml
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[yonzhan]

route to CXP team

[navba-MSFT]

@roalexan Apologies for the late reply. Could you please test the deployment again and let me know if you are still facing the issue ? Awaiting your reply.

[spacedragon]

@roalexan Could you please update the azureml-defaults to version 1.38.0? This should fix your problem.

channels:
  - conda-forge
dependencies:
  - python==3.7.7
  - pip>=19.0.3
  - pip:
      - 'azureml-sdk[notebooks,contrib]==1.32.0'
      - azureml-defaults==1.38.0       
      - spacy==3.0.4
      - joblib==1.1.0

[robotNo201]

I have the same problem as well, I try the solution above, but it doesn't work. I tried deploying my model before with a different yml file and it works, but I deleted some dependencies in yml file yesterday for a nice looking, and I got this error now.

[roalexan]

out for afternoon - will test when back this evening.

[roalexan]

operation.log - link here - shows a pip error:

...
Pip subprocess error:
ERROR: Cannot install -r /azureml-environment-setup/condaenv.oi9s838c.requirements.txt (line 2) and azureml-sdk[contrib,notebooks]==1.32.0 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

CondaEnvException: Pip failed

�[0mThe command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_aedc9ab9f824c610cb68a40af0619900 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name pycache -exec rm -rf {} + && ldconfig' returned a non-zero code: 1
2022/02/21 22:58:41 Container failed during run: acb_step_0. No retries remaining.
failed to run step ID: acb_step_0: exit status 1

Run ID: ca1 failed after 3m0s. Error: failed during run, err: exit status 1

[roalexan]

maybe 'azureml-sdk[notebooks,contrib]==1.32.0' needs to be increased to 1.38.0 as well? I'm going to try that.

[roalexan]

think that got further, but now see this error:

...
main_module_spec.loader.exec_module(main)
File "/var/azureml-app/src/score_pretrained.py", line 9, in
from utils import get_matches_dict, get_entity_counts, get_loc_selected, get_coordinates
File "/var/azureml-app/src/utils.py", line 10, in
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
Worker exiting (pid: 40)
Shutting down: Master
Reason: Worker failed to boot.
2022-02-21T23:31:30,621868765+00:00 - gunicorn/finish 3 0
2022-02-21T23:31:30,623743794+00:00 - Exit code 3 is not normal. Killing image.

Well, at least this seems to be in the source code of our scoring script, so I will confer with the author of this tomorrow and report back.

[navba-MSFT]

@roalexan Thanks for getting back and sharing the error details. The workspace link you had shared is not accessible to me due to permission issue. Moreover the error message is from your script due to the missing Pandas module.
There are similar issues reported for the same. You can share this and this article with the script author if that helps.

[sergey-ivanchuk]

I'm dealing with the same situation myself. Any formal feedback & position from Microsoft would be great here.

So far I've been downgrading to itsdangerous==2.0.1 to resolve the new bug. The release notes for 2.1.0 seem pretty minor, and don't seem to meaningfully modify the core package. However the tweak around importing JSON is obviously causing issues for AML. I would be very surprised if Flask needs 2.1.0 for anything

[MikeLamoreux]

Can someone explain how to downgrade to "itsdangerous==2.0.1"? I started using AzureML this week and can't even complete trainings. I've tried creating an environment and putting the value in a yaml file. I've also tried to do the CondaDependencies add_conda_package as well as add_pip_package. It still always fails and in the deployment logs I see the PIP Dependencies still show "itsdangerous=2.1.0".

[roalexan]

temporary conda env workaround I'm using (bolded line added):

            - pip:
                - 'azureml-sdk[notebooks,contrib]==1.32.0'
                - azureml-defaults==1.32.0
                - spacy==3.0.4
                - joblib==1.1.0
                - **itsdangerous==2.0.1**

[navba-MSFT]

This problem should be fixed already, could you please use the the latest version of python packages ?
The version of azureml-defaults package in your conda file should be - azureml-defaults=1.38.0
Also please make sure you are using a newly created environment instead of reusing the old before the fix. Also refer this. Hope this helps.

[roalexan]

Yes, I agree that updating from 1.32.0 to 1.38.0 fixes this particular problem, and perhaps you can close this bug.

Just please be aware that this approach, in my environment, introduces another error in that 'pandas' cannot be found. I can probably get that to work, but for now have opted to stick with 1.32.0 and add 'itsdangerous==2.0.1' until I have more time to fully figure this out.

[navba-MSFT]

@roalexan Thanks for getting back. We will now proceed with closure of this github issue. If this issue resurfaces in future, Please let us know. We would be happy to help.