Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msrest and msrestazure dependencies not installed #644

Closed
freefood89 opened this issue Jun 23, 2016 · 38 comments
Closed

msrest and msrestazure dependencies not installed #644

freefood89 opened this issue Jun 23, 2016 · 38 comments
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Packaging

Comments

@freefood89
Copy link

Is it just me, or does this package not install dependencies?

  1. installed via pip3 install azure
    from azure.common.credentials import UserPassCredentials resulted in ImportError: No module named 'msrest'
  2. installed msrest via pip3 install msrest
    from azure.common.credentials import UserPassCredentials resulted in ImportError: No module named 'msrestazure'
  3. installed msrest azure via pip3 install msrestazure
    from azure.common.credentials import UserPassCredentials finally worked
@lmazuel
Copy link
Member

lmazuel commented Jun 23, 2016

Hi @freefood89 ,

I saw rare situations where msrest(azure) dependencies are not installed. I think it's related to this pip issue:
pypa/pip#988

Because msrest(azure) in a sub-dependencies of an azure-common extension called autorest. But since some other package does not need the extension, it seems that if pip sees an azure-common dependency before it sees azure-common[autorest] dependency, it does not install the extension.

Thank you for the feedback, I will try to find a smart way to make this more predictable.

@elemantalcode
Copy link

I seem to have the same problem. I initially did: pip install azure==2.0.0rc5 and tried to run https://github.com/ansible/ansible/blob/devel/contrib/inventory/azure_rm.py but get:

[terraform@pc001 ~]$ ./azure_rm.py
The Azure python sdk is not installed (try 'pip install azure==2.0.0rc5') - No module named msrestazure.azure_exceptions

[terraform@pc001 ~]$ pip install azure==2.0.0rc5
Requirement already satisfied (use --upgrade to upgrade): azure==2.0.0rc5 in /usr/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): azure-storage==0.32.0 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-servicebus==0.20.2 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-servicemanagement-legacy==0.20.3 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-batch==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-graphrbac==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-nspkg in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): futures in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-common in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-scheduler==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-compute==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-keyvault==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-notificationhubs==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-cdn==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-powerbiembedded==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-redis==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-commerce==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-cognitiveservices==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-resource==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-storage==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-authorization==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-batch==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-network==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-logic==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-web==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil->azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-nspkg in /usr/lib/python2.7/site-packages (from azure-mgmt-scheduler==0.30.0rc5->azure-mgmt==0.30.0rc5->azure==2.0.0rc5)

[terraform@pc001 ~]$ pip install msrestazure
Requirement already satisfied (use --upgrade to upgrade): msrestazure in /usr/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): msrest>=0.4.0 in /usr/lib/python2.7/site-packages (from msrestazure)
Requirement already satisfied (use --upgrade to upgrade): chardet>=2.3.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): certifi>=2015.9.6.2 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): enum34>=1.0.4 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): keyring>=5.6 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): isodate>=0.5.4 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.7.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): requests-oauthlib>=0.5.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): secretstorage; sys_platform == "linux2" or sys_platform == "linux" in /usr/lib/python2.7/site-packages (from keyring>=5.6->msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): oauthlib>=0.6.2 in /usr/lib/python2.7/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): pycrypto in /usr/lib64/python2.7/site-packages (from secretstorage; sys_platform == "linux2" or sys_platform == "linux"->keyring>=5.6->msrest>=0.4.0->msrestazure)


mrestazure was not installed as a dependency during the initial install of pip install azure

Some versioning:
OS: Red Hat Enterprise Linux Server release 6.8 (Santiago)
PIP: pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)

Please note @lmazuel The above post is simply to showcase and inform that installing azure does not include all dependencies in specific situations as you've previously observed; the contrib script is out of scope for this section. (The fix was to point the contrib script to python 2.7 instead of 2.6).

@lmazuel
Copy link
Member

lmazuel commented Aug 8, 2016

Thanks for the feedback @elemantalcode !

@grindthemall
Copy link

Just ran into this same issue.

"pip install msrestazure" corrected the problem for me.

OS: Centos 7.2
PIP: 8.1.2
Python 2.7.5

@lmazuel lmazuel added Packaging bug This issue requires a change to an existing behavior in the product in order to be resolved. labels Sep 14, 2016
@xingzhougmu
Copy link

xingzhougmu commented Oct 11, 2016

@lmazuel
I install Azure Python SDK from the git source, but running into the same issue when "import azure.mgmt.redis"

I use the following commands to install SDK for Python3.5
git clone git://github.com/Azure/azure-sdk-for-python.git
cd azure-sdk-for-python
python3 setup.py install

My OS is: Ubuntu 16.04LTS

@trondhindenes
Copy link

This issue is now 6 months old, and there's still not a clean way to install the azure sdk for python. Any updates on this?

@squasta
Copy link

squasta commented Jan 9, 2017

Hi, I have also problems with dependencies with Ansible (running Ubuntu 16.04 LTS) with Azure SDK for python. I tried everything described on this page. nothing works. Still get the message Try pip install azure==2.0.0rc5- No module named common.credentials

@lmazuel
Copy link
Member

lmazuel commented Jan 9, 2017

@trondhindenes I started to change the way I install msrestazure, replacing azure-common[autorest] that creates the problem by an explicit msrestazure dependency, see for instance this setup.py file. Each time I release a new package, I take this opportunity to update it. I thought it was too soon to answer here, but I didn't forget and eventually all packages will be fixed. This is NOT a fix in the azure meta-package, since as a meta-package it does not have a dependency on msrestazure.

@squasta if you have a message "no module named common.credentials", then you're not into this issue. This issue installs correctly azure.common.credentials BUT doesn't install the underlying package to make it work, so you got a msrestazure error that can be fixed installing manually msrestazure. If you still think you have the issue mentioned here, please provide a stackstrace. If not, please create a new issue with your specific stacktrace.

@trondhindenes
Copy link

Thanks for the update @lmazuel , good to know that the issue is being worked on.

@lmazuel
Copy link
Member

lmazuel commented Jan 10, 2017

@trondhindenes I pushed a commit and changed them all:
3dc2d4f

GH has closed the issue automatically, I guess it's fair, but the fix will be published package per package only each time I publish new release. I will not publish new version of everything just for this. However, an update of azure-mgmt-compute and azure-mgmt-resource is planned for early February, so since they are the most used packages, problem will disappear quickly.

Please feel free to still comment here, even closed, and I re-open it if necessary.

@scmdavid
Copy link

scmdavid commented Feb 20, 2017

I am using Mac (10.12.3) and python 2.7.13
I manually run
sudo pip install msrest (0.4.5)
sudo pip install msrestazure (0.4.7)
sudo pip install "azure==2.0.0rc5"

All succeed but I still got :
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure==2.0.0rc5 installed? Try pip install azure==2.0.0rc5- No module named msrest.serialization"}

@rkumar357
Copy link

I have the same issue , any update on this ?

@lmazuel
Copy link
Member

lmazuel commented Mar 2, 2017

Hi @rkumar357
As I wrote in my latest message, new packages are now released with the fix. I will not release a new package just for that issue, but it will be fixed by all new package and eventually in every packages.
Thanks,

@rkumar357
Copy link

I have problem with Python 2.7.10 , Python 2.7.13 got succeeded.

@lmazuel
Copy link
Member

lmazuel commented Mar 2, 2017

It's a random issue, depending on the order pip process the dependencies. This is not related to Python version (unless Python is shipped with pip and pip is newer with 2.7.13, it's possible).
New packages released since 01/2017 does not have the issue. I cannot fix the old ones, where you might have to pip install msrestazure manually.
thanks,

@rkumar357
Copy link

Cheers @lmazuel , working fine , thanks for the help.

@ghost
Copy link

ghost commented Mar 3, 2017

So I just installed azure az new installation on fedora 24, got the same error and cannot use module at all. Any advise ?

Same in docker ubuntu after : pip install "azure==2.0.0rc5"
python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from azure.common.credentials import UserPassCredentials
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named azure.common.credentials

@lmazuel
Copy link
Member

lmazuel commented Mar 3, 2017

Hi @Ment0s
You don't have the same error, since the missing module is azure.common.credentials, not msrestazure. Could you provide a more detailled description of the messages you got from pip during the installation?

@ghost
Copy link

ghost commented Mar 4, 2017

pip output : http://pastebin.com/RgQDNF0s

With ansible from git I get these errors :

"Do you have azure==2.0.0rc5 installed? Try pip install azure==2.0.0rc5- cannot import name integer_types"

pip install rc5 : http://pastebin.com/QNrsFk6Y

@rkumar357
Copy link

@Ment0s , ansible using old version of python, just check in verbose mode, to execute any pthyon script ansible will use /usr/bin/python or something esle ?

/usr/bin/python --version
python --version
/usr/bin/lib/python --version

if you have done brew install then Cellar/bin/python --version

Compare which python is used while executing ansible python script, i have soft link 2.7 python library to /usr/bin/python, since ansible is using that location for me.

@mkesselaers
Copy link

I have the same problem with a fresh install on MacOS
TASK [Create a resource group] ************************************************* fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have msrestazure installed? Try pip install msrestazure- No module named msrest.serialization"}

Python version is 2.7.13.
My pip list is the following (only relevant packages).
azure (1.0.3) azure-batch (0.30.0rc5) azure-common (1.1.4) azure-graphrbac (0.30.0rc5) azure-mgmt (0.20.2) azure-mgmt-authorization (0.30.0rc5) azure-mgmt-batch (0.30.0rc5) azure-mgmt-cdn (0.30.0rc5) azure-mgmt-cognitiveservices (0.30.0rc5) azure-mgmt-commerce (0.30.0rc5) azure-mgmt-common (0.20.0) azure-mgmt-compute (0.20.1) azure-mgmt-keyvault (0.30.0rc5) azure-mgmt-logic (0.30.0rc5) azure-mgmt-network (0.20.1) azure-mgmt-notificationhubs (0.30.0rc5) azure-mgmt-nspkg (2.0.0) azure-mgmt-powerbiembedded (0.30.0rc5) azure-mgmt-redis (0.30.0rc5) azure-mgmt-resource (0.20.1) azure-mgmt-scheduler (0.30.0rc5) azure-mgmt-storage (0.20.0) azure-mgmt-web (0.30.0rc5) azure-nspkg (2.0.0) azure-servicebus (0.20.1) azure-servicemanagement-legacy (0.20.2) azure-storage (0.20.3) msrest (0.4.7) msrestazure (0.4.7)

I tried uninstalling and re-installing azure, azure-mgmt, msrest and msrestazure, but this didn't work.
Any idea about what is wrong?

@lmazuel
Copy link
Member

lmazuel commented Apr 24, 2017

Hi @mkesselaers

Can you open a python interpreter and type "import msrest"?

Thanks,

@mkesselaers
Copy link

Hi @lmazuel,

This doesn't give any error, which seems like everything is fine.

Best regards,
Maarten

@lmazuel
Copy link
Member

lmazuel commented Apr 24, 2017

If you are able to import msrest from a Python interpreter but Ansible can't, might be a problem with Ansible (it's Ansible right?)

@mkesselaers
Copy link

Yes, you're right. But since the error that I've got was found on this thread, I thought that it would be best to post it here.
How did the other OPs solved it?

@rkumar357
Copy link

Try running,

ansible-playbook test.yml -e ansible_python_interpreter=/usr/local/opt/python/bin/python2.7

@mkesselaers
Copy link

Hi @rkumar357 ,

This works indeed.
I presume that Ansible now defaults to python 3 as interpreter?

Thanks for your help!
Best regards,
Maarten

@rkumar357
Copy link

Cheers !!! for ansible i belive python 2 is deafult.

@lmazuel
Copy link
Member

lmazuel commented Apr 25, 2017

@rkumar357 @mkesselaers Ansible does not support Python 3 at all for now (see on PyPI). So only option here is Python 2.7. I don't know how Ansible is defining his shebang, but if for some reason it starts with Py3 by default you will be in trouble.
I didn't know Ansible has a ansible_python_interpreter (thanks @rkumar357 :)). I will keep this in mind and that's indeed the correct solution.

@aaronjones
Copy link

aaronjones commented May 1, 2017

I am running Ansible (2.2.2.0 or 2.3.0.0) inside a docker container and am receiving the same msrestazure.azure_exceptions error despite applying the various fixes noted above.

ansible-playbook -vvvv  -i /home/ansible/ansible/contrib/inventory/azure_rm.py generatevm.yml -e ansible_python_interpreter=/usr/local/opt/python/bin/python2.7
...
ERROR! Attempted to execute "/home/ansible/ansible/contrib/inventory/azure_rm.py" as inventory script: Inventory script (/home/ansible/ansible/contrib/inventory/azure_rm.py) had an execution error: The Azure python sdk is not installed (try `pip install 'azure>=2.0.0rc5' --upgrade`) - No module named msrestazure.azure_exceptions
pip show azure
Name: azure
Version: 2.0.0rc6
Summary: Microsoft Azure Client Libraries for Python
Home-page: https://github.com/Azure/azure-sdk-for-python
Author: Microsoft Corporation
Author-email: ptvshelp@microsoft.com
License: MIT License
Location: /usr/local/lib/python2.7/dist-packages

Looking at the azure_rm.py code, it looks like it is just failing upon import (I'm not a python dev... yet). From what I have read, providing the reference to the path where azure_rm.py lives should be sufficient for python to pick up the remaining libraries listed in the import... though I'm not certain.
I cloned https://github.com/Azure/azure-sdk-for-python.git on Friday, so that shouldn't be out of date, and I've tried both rc5 & rc6 pip installs of azure .
Any tips would be appreciated.

@lmazuel
Copy link
Member

lmazuel commented May 1, 2017

Hi @aaronjones

Could you try:

/usr/local/opt/python/bin/python2.7 -c "import msrestazure; print(msrestazure.__version__)"

@aaronjones
Copy link

slightly embarrassing, but delivers the same result and same problem. (I didn't realize python wasn't installed in that same location, so it must have defaulted to what was installed)
Using the following command:
/usr/bin/python2.7 -c "import msrestazure; print(msrestazure.version)"
results in:

Traceback (most recent call last):
File "", line 1, in
ImportError: No module named msrestazure

@lmazuel
Copy link
Member

lmazuel commented May 1, 2017

So, you can install msrestazure this way (assuming you have sudo access):
sudo /usr/bin/python2.7 -m pip install msrestazure

@aaronjones
Copy link

Sorry for the delay in response. It has at least moved me forward to a credentials error, so that is progress. I guess the ansible documentation or possibly the sdk readme may need updating to reflect the requirement of installing mrestazure separately?
https://docs.ansible.com/ansible/guide_azure.html
I'm guessing this is just an issue running ansible inside of docker given other positive responses above?

@lmazuel
Copy link
Member

lmazuel commented May 2, 2017

msrestazure is not supposed to be installed separately, but we got issues in the past with pip. We fixed that in recent packages, but likely you got it with old version :(
Installing the future azure 2.0.0rc7 should definitely fix the issue by forcing all packages to be in a recent version where the issue is fixed.

@dselans
Copy link

dselans commented Jun 5, 2017

I just ran into this on OS X 10.11.6. Just in case someone else runs into this, my (seemingly) underlying issue was that I installed ansible via brew, which installs python 2.7.13 as a dep. Running pip install azure* via local python 2.7.10 ended with the ImportError: No module named 'msrestazure' issue.

The fix was to ensure that I'm using python 2.7.13 - I went ahead and updated my $PATH to include /usr/local/Cellar/python/2.7.13/bin. After this, running sudo pip install azure==2.0.0rc5 succeeded.

@redice
Copy link

redice commented Aug 10, 2018

My solution is a workable way to solve No module named msrest.serialization issue.
In OS X 10.13.6, run brew instabll ansible to install the latest ansbile package (my case is 2.6.2)
Ansible 2.6.2 has its own python 2.7.15 in /usr/local/Cellar/ansible/2.6.2/libexec/bin directory

Need to use its pip script to install azure packages
/usr/local/Cellar/ansible/2.6.2/libexec/bin/pip install azure to fix the azure dependences.

@kb185177
Copy link

kb185177 commented Jun 4, 2022

I have a similar issue, not sure if its a compatibility issue between msrest version and azure commons , tried reinstalling, installling on a new venv. I am trying to use azure azure_rm_keyvault_info - Here is the error trace - Any suggestions would help >

  File "/var/folders/yh/9flcl8g13pd0fvdwq3_vpnb9t15tdk/T/ansible_azure_rm_keyvault_info_payload_90lcafqh/ansible_azure_rm_keyvault_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 222, in <module>
    from msrest.serialization import Serializer
ModuleNotFoundError: No module named 'msrest'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {

azure-keyvault-secrets 4.4.0 requires msrest>=0.6.21, but you have msrest 0.4.29 which is incompatible.

azure-mgmt-keyvault 0.30.0rc6 requires azure-common[autorest]==1.1.4, but you have azure-common 1.1.28 which is incompatible.

tried installing them manually
pip install "azure-common[autorest]==1.1.4"
pip install "msrest==0.6.21"

no luck

@github-actions github-actions bot locked and limited conversation to collaborators Apr 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Packaging
Projects
None yet
Development

No branches or pull requests