Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to use Azure ARM cloud provider with salt cloud on salt 3002 #59264

Closed
martinhacker1965 opened this issue Jan 9, 2021 · 5 comments
Closed

Unable to use Azure ARM cloud provider with salt cloud on salt 3002 #59264

martinhacker1965 opened this issue Jan 9, 2021 · 5 comments
Labels
Bug broken, incorrect, or confusing behavior Salt-Cloud severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Milestone
Approved

Comments

@martinhacker1965
Copy link

Description of Issue

Attempting to perform a POC for provisioning Azure virtual machines using the salt cloud azurearm driver from a salt master also running in Azure. Master is running on Ubuntu 20_04-lts. See Setup section for inital bootstrap configuration.

Referred to https://docs.saltstack.com/en/latest/topics/cloud/azurearm.html to check dependencies. These refer to sdk libraries that are now deprecated such as azure and azure-storage - see https://pypi.org/project/azure/ where the following is stated:
Please install the service specific packages prefixed by azure needed for your application.
The same dependencies are also listed in /usr/lib/python3/dist-packages/salt/cloud/clouds/azurearm.py

Following guidance on using azure python sdk libraries at https://docs.microsoft.com/en-gb/python/api/overview/azure/virtualmachines?view=azure-python-preview I decided to install a few libraries which might be required to create an Azure VM then began testing.

pip3 install -U azure-mgmt-compute azure-mgmt-network azure-mgmt-resource azure-mgmt-storage azure-mgmt-web azure-mgmt-authorization

Setup

#salt-master.sh

#! /bin/bash
wget -O - https://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -

echo "
deb http://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest focal main
" | sudo tee /etc/apt/sources.list.d/saltstack.list

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install linux-azure linux-cloud-tools-azure linux-headers-azure linux-image-azure linux-tools-azure -y
sudo apt-get install salt-master salt-cloud -y
sudo apt install python3-pip -y
pip3 install -U azure-mgmt-compute azure-mgmt-network azure-mgmt-resource azure-mgmt-storage azure-mgmt-web

Steps to Reproduce Issue

  1. Created a new service principal in destination subscription.

  2. Created /etc/salt/cloud.providers.d/azurearm.conf with the following contents

azurearm-conf:
  driver: azurearm
  subscription_id: mysubscription_id
  client_id: myclent_id
  secret: mysecret
  tenant: mytenant
  minion:
    master: mymasterurl
  1. Created /etc/salt/cloud.profiles.d/azurearm.conf with the following contents
azure-centos:
  provider: azurearm-conf
  image: cognosys|centos-8-2|centos-8-2|latest
  size: Standard_A2_v2
  location: uksouth
  ssh_username: myusername
  ssh_password: mypassword
  1. Ran the following:
sudo systemctl restart salt-master
  1. Ran the following:
sudo salt-cloud -p azure-centos test1 -l debug

Got following output:
[DEBUG ] Reading configuration from /etc/salt/cloud
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/azurearm.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/azurearm.conf
[DEBUG ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/azurearm.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/azurearm.conf
[DEBUG ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO ] salt-cloud starting
[WARNING ] Missing dependency: 'azurearm'. The azurearm driver requires 'azurearm' to be installed.
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha1' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'hmac_compute' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'indent' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Marking 'method_call' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[DEBUG ] Could not LazyLoad parallels.avail_sizes: 'parallels' virtual returned False
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] LazyLoaded azurearm.get_configured_provider
[WARNING ] The cloud driver, 'azurearm', configured under the 'azurearm-conf' cloud provider alias, could not be loaded. Please check your provider configuration files and ensure all required dependencies are installed for the 'azurearm' driver.
In rare cases, this could indicate the 'azurearm.get_configured_provider()' function could not be found.
Removing 'azurearm' from the available providers list
[ERROR ] There was a profile error: 'azurearm-conf'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/cloud/cli.py", line 290, in run
ret = mapper.run_profile(self.options.profile, self.config.get("names"))
File "/usr/lib/python3/dist-packages/salt/cloud/init.py", line 1392, in run_profile
provider_details = self.opts["providers"][alias][driver].copy()
KeyError: 'azurearm-conf'

  1. Ran the following:
sudo salt-cloud --list-images azurearm-conf -l debug

Got following output:
[DEBUG ] Reading configuration from /etc/salt/cloud
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/azurearm.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/azurearm.conf
[DEBUG ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/azurearm.conf'
[DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/azurearm.conf
[DEBUG ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO ] salt-cloud starting
[WARNING ] Missing dependency: 'azurearm'. The azurearm driver requires 'azurearm' to be installed.
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha1' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'hmac_compute' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'indent' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Marking 'method_call' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[DEBUG ] Could not LazyLoad parallels.avail_sizes: 'parallels' virtual returned False
[DEBUG ] LazyLoaded parallels.avail_locations
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] LazyLoaded azurearm.get_configured_provider
[WARNING ] The cloud driver, 'azurearm', configured under the 'azurearm-conf' cloud provider alias, could not be loaded. Please check your provider configuration files and ensure all required dependencies are installed for the 'azurearm' driver.
In rare cases, this could indicate the 'azurearm.get_configured_provider()' function could not be found.
Removing 'azurearm' from the available providers list
Error: There was an error listing images: No cloud providers matched 'azurearm-conf'. Available selections:

Versions Report

Salt Version:
Salt: 3002.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.5 (default, Jul 28 2020, 12:59:40)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2

System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-1031-azure
system: Linux
version: Ubuntu 20.04 focal
@welcome
Copy link

welcome bot commented Jan 9, 2021

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at core@saltstack.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

@sagetherage sagetherage added Bug broken, incorrect, or confusing behavior needs-triage labels Jan 11, 2021
@twangboy
Copy link
Contributor

Looks like the azure cloud driver needs some love.

@twangboy twangboy added this to the Approved milestone Jan 14, 2021
@twangboy twangboy removed their assignment Jan 14, 2021
@martinhacker1965
Copy link
Author

Hoping someone can pick this up. We are currently using Saltstack to manage virtual machines on all of our internal platforms and now want to extend this to Azure.

@nicholasmhughes
Copy link
Collaborator

@martinhacker1965 this is probably breaking here: https://github.com/saltstack/salt/blob/master/salt/cloud/clouds/azurearm.py#L129

MS has significantly changed their APIs (and SDKs) since late 2017 / early 2018 when this was written. I seem to recall that the storage SDK changed enough that this import won't even work.

Try installing these older SDK versions and see if that helps:
#54767 (comment)

Additionally, you might want to check out https://github.com/eitrtechnologies/idem-azurerm

It's the "next gen" Azure provider. Not fully integrated w/ Salt, but might be useful for some workflows.

Hit me on the community Slack if you need help with any of that.

@sagetherage sagetherage added severity-high 2nd top severity, seen by most users, causes major problems severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around and removed severity-high 2nd top severity, seen by most users, causes major problems labels Jun 15, 2021
@nicholasmhughes
Copy link
Collaborator

The Azure functionality in Salt is moving to https://github.com/salt-extensions/saltext-azurerm

Please try out the extension and open an issue there if you are still encountering problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug broken, incorrect, or confusing behavior Salt-Cloud severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Projects
None yet
Milestone
Approved
Development

No branches or pull requests
4 participants
@nicholasmhughes @twangboy @martinhacker1965 @sagetherage