Collection's requirements-azure.txt packages refer to conflicting versions for cli authentication with azure-cli

[octagons]

SUMMARY

The az command does not get installed by the collection, therefore we need to install the azure-cli package. However, the packages installed by the collection's requirements-azure.txt file conflict with recent versions of the package azure-cli, including 2.34.0 and 2.46.0.

We are trying to stick to using the cli auth source, hence our need for the az command in order to run az login.

If possible, please review the following:

• Is cli-based authentication still considered a viable method by which to use this collection?
• What is the appropriate package installation process to use cli-based authentication for this collection?

ISSUE TYPE

• Bug Report

COMPONENT NAME

plugins/module_utils/azure_rm_common.py

ANSIBLE VERSION

ansible [core 2.14.4]
  config file = /home/user/test_dir/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/test_dir/lib/python3.10/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/test_dir/bin/ansible
  python version = 3.10.10 (main, Mar  5 2023, 22:26:53) [GCC 12.2.1 20230201] (/home/user/test_dir/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/user/test_dir/lib/python3.10/site-packages/ansible_collections
Collection         Version
------------------ -------
azure.azcollection 1.15.0 

CONFIGURATION

CONFIG_FILE() = /etc/ansible/ansible.cfg

OS / ENVIRONMENT

• Arch Linux - latest as of date of issue (i.e. pacman -Syu --noconfirm && reboot)
• Virtual machine running on Windows
• Installing all dependencies in a Python virtual environment
• This workflow is part of a larger playbook which creates infrastructure and updates the inventory
• This process assumes you have an active subscription with a load balancer configured with dynamically allocated public IPv4 addresses
• The steps below are an isolated way to reproduce the issue

STEPS TO REPRODUCE

1. Install OS Python 3 package venv (usually found as python3-venv)
2. Create a virtual environment in a directory: python3 -m venv test_dir
3. Change directory to test_dir, then activate the virtual environment: source bin/activate
4. Upgrade pip to the latest version: pip install --upgrade pip
5. Then create a requirements.txt file with the following contents (requirements for our project):

ansible
ansible-core
bcrypt
cryptography

6. Update the $PATH with the newly install binaries: deactivate && source bin/activate
7. Install packages from the requirements.txt file: pip install -r requirements.txt
8. Create a requirements.yml file with the following contents (collection requirements for our project):

---
collections
  - azure.azcollection
  - community.crypto
  - community.general
  - community.postgresql

9. Install packages from the collection's requirements file (the Python version might differ): pip install -r lib/python3.10/site-packages/ansible_collection/azure/azcollection/requirements-azure.txt
10. Run this command and note the lack of an az command in $PATH: which az
11. Install the azure-cli package (affected versions include at least 2.34.0 and 2.46.0): pip install "azure-cli==2.34.0"
12. Login to Azure: az login
13. Configure your subscription ID to one with a load balancer/public IPs: az account set -s <subscription_id>
14. Confirm that you have a load balancer: az network lb list
15. Create a configuration file for the azure_rm inventory plugin named test_config.azure_rm.yml:

---
plugin: azure.azcollection.azure_rm
auth_source: cli

16. Attempt to query information about the load balancer with the -_info module from the collection: ansible localhost -i test_config.azure_rm.yml -m azure.azcollection.azure_rm_loadbalancer_info
17. Note the error.
18. Note that the inventory plugin appears to be unaffected: ansible-inventory -i test_config.azure_rm.yml --list

EXPECTED RESULTS

I expect the module to return a list of resources specified by the module. In this case, a list of load balancers and their configuration information in a JSON-encoded format.

ACTUAL RESULTS

The module fails to import a class from azure.mgmt.recoveryservicesbackup because this class has been moved to a new namespace in the Azure SDK for Python. Based on steps above, version 5.2.0 is installed for package azure-mgmt-recoveryservicesbackup. In this version, the RecoveryServicesBackupClient class has been moved to the namespace azure.mgmt.recoveryservicesbackup.activestamp. See: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/recoveryservices/azure-mgmt-recoveryservicesbackup/azure/mgmt/recoveryservicesbackup/activestamp/__init__.py.

Note that several other modules are affected in the same way, such as azure.azcollection.azure_rm_publicipaddress_info.

See below for the error message produced by Ansible:

$ ansible localhost -i test_config.azure_rm.yml -m azure.azcollection.azure_rm_loadbalancer_info
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' (/home/user/test_dir/lib/python3.10/site-packages/azure/mgmt/recoveryservicesbackup/__init__.py)
localhost | FAILED! => {
    "changed": false,
    "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on test_hosts's Python /home/user/test_dir/bin/python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"
}

[octagons]

It looks like the following order of operations works for our intent, but I believe this is still a regression.

• pip install -r requirements.txt
• ansible-galaxy collection install -r requirements.yml
• pip install "azure-cli==2.34.0"
• pip install -r lib/python3.10/site-packages/ansible_collections/azure/azcollection/requirements-azure.txt

[ygo74]

https://pypi.org/project/azure-cli/#history
version 2.34.0 has been released one year ago !

As all features are not implemented in the collection, it is important to be able to have the two tools installed together

Change order doesn't work for me. Installation is done in a docker alpine image to have the same tools on my computer and in the azure devops pipeline

FROM node:14-alpine

RUN apk update && apk upgrade \
    && apk add --no-cache --virtual .pipeline-deps readline linux-pam \
    && apk add bash sudo shadow \
    && apk del .pipeline-deps

LABEL "com.azure.dev.pipelines.agent.handler.node.path"="/usr/local/bin/node"

RUN apk update && apk upgrade \
    && apk add --no-cache --virtual .pipeline-deps readline linux-pam \
    && apk add bash sudo shadow \
    && apk add bash py3-pip \
    && apk add --virtual=build gcc libffi-dev musl-dev openssl-dev python3-dev make \
    && pip --no-cache-dir install -U pip \
    && pip install -r https://raw.githubusercontent.com/ansible-collections/azure/v1.15.0/requirements-azure.txt \
    && pip install ansible==7.3.0 \
    && ansible-galaxy collection install azure.azcollection:1.15.0 \
    && pip install kubernetes==26.1.0 \
    && ansible-galaxy collection install kubernetes.core:2.4.0 \
    && pip install azure-cli==2.34.0 \
    && pip install -r /usr/lib/python3.10/site-packages/ansible_collections/azure/azcollection/requirements-azure.txt \
    && apk del .pipeline-deps \
    && apk del --purge build

CMD [ "node" ]

By changing the order, I have a new error:

fatal: [iasaksbootstrap]: FAILED! => {"changed": false, "msg": "Error retrieving resource group rg-acr-bootstrap - ResourceManagementClient.__init__() got an unexpected keyword argument 'credential'"}

[Fred-sun]

@octagons Since there will be conflict between azure-cli and requirements-azure.txt when installed together, it is recommended to directly install the specified version of azure-cli to effectively avoid this problem, thank you!

Like this:
1. sudo pip3 install ansible
2. ansible-galaxy collection install azure.azcollection
3. pip3 install azure-cli==2.34.0
4. pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

Or: 
1. sudo pip3 install ansible
2. ansible-galaxy collection install azure.azcollection
3. pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
4. pip3 install azure-cli==2.34.0