Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt-cloud azurearm is not working #49883

Open
kiemlicz opened this issue Oct 3, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@kiemlicz
Copy link
Contributor

commented Oct 3, 2018

Description of Issue/Question

I'm unable to configure and do anything via salt-cloud azurearm cloud provider

Setup

I've followed https://docs.saltstack.com/en/latest/topics/cloud/azurearm.html
cloud.providers.d/azure.conf

some_azure:
  driver: azurearm
  subscription_id: "lashfoasuhbaeiruhvklhf" #edited
  username: "username@mydomain"
  password: "blabla"
  location: westeurope
  resource_group: somegroup

Steps to Reproduce Issue

salt-cloud --list-locations some_azure
yields:

[WARNING ] The required 'tenant' configuration setting is missing from the 'azurearm' driver, which is configured under the 'some_azure' alias.
No minions matched the target. No command was sent, no jid was assigned.
No minions matched the target. No command was sent, no jid was assigned.
[WARNING ] The required 'tenant' configuration setting is missing from the 'some_azure:azurearm' driver, which is configured under the 'some_azure' alias.
[WARNING ] The required 'tenant' configuration setting is missing from the 'some_azure:azurearm' driver, which is configured under the 'some_azure' alias.
[WARNING ] The required 'tenant' configuration setting is missing from the 'some_azure:azurearm' driver, which is configured under the 'some_azure' alias.
[WARNING ] The required 'tenant' configuration setting is missing from the 'some_azure:azurearm' driver, which is configured under the 'some_azure' alias.
[WARNING ] The required 'tenant' configuration setting is missing from the 'some_azure:azurearm' driver, which is configured under the 'some_azure' alias.
[ERROR   ] Failed to get the output of 'azurearm.avail_locations()': 'WebSiteManagementClient' object has no attribute 'global_model'

listing sizes, images - similar result e.g.

[ERROR   ] Failed to get the output of 'azurearm.avail_sizes()': Parameter 'location' can not be None.

EDIT:
after upgrading azure-cli I'm receiving:

[ERROR   ] Failed to get the output of 'azurearm.avail_sizes()': , AdalError: Get Token request returned http error: 400 and server response: {"error":"invalid_request","error_description":"AADSTS90014: The request body must contain the following parameter: 'client_id'.\r\nTrace ID: #edited}

I don't see a thing in doc about client_id (either way setting it to username forces me to set secret, tenant etc that I don't have)

Versions Report

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: 1.10.6
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: 0.27.4
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.27.1
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.5
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-8-amd64
         system: Linux
        version: debian 9.5
@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2018

@nicholasmhughes can you help with this?

It looks like the tenant configuration missing is fine, because there are two ways to configure this, and the other way is setup correctly.

Thanks,
Daniel

@nicholasmhughes

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2018

@kiemlicz can you dump the Azure SDK versions? pip freeze | egrep 'azure|msrest'

@kiemlicz

This comment has been minimized.

Copy link
Contributor Author

commented Oct 3, 2018

Thank you for response, please find the output:

azure==4.0.0
azure-applicationinsights==0.1.0
azure-batch==5.0.0
azure-cli==2.0.46
azure-cli-acr==2.1.5
azure-cli-acs==2.3.4
azure-cli-advisor==0.6.0
azure-cli-ams==0.2.3
azure-cli-appservice==0.2.4
azure-cli-backup==1.2.1
azure-cli-batch==3.4.0
azure-cli-batchai==0.4.3
azure-cli-billing==0.2.0
azure-cli-cdn==0.1.1
azure-cli-cloud==2.1.0
azure-cli-cognitiveservices==0.2.3
azure-cli-command-modules-nspkg==2.0.2
azure-cli-component==2.0.8
azure-cli-configure==2.0.18
azure-cli-consumption==0.4.0
azure-cli-container==0.3.4
azure-cli-core==2.0.46
azure-cli-cosmosdb==0.2.1
azure-cli-dla==0.2.3
azure-cli-dls==0.1.3
azure-cli-dms==0.1.1
azure-cli-eventgrid==0.2.0
azure-cli-eventhubs==0.2.4
azure-cli-extension==0.2.1
azure-cli-feedback==2.1.4
azure-cli-find==0.2.12
azure-cli-interactive==0.3.30
azure-cli-iot==0.3.2
azure-cli-keyvault==2.2.3
azure-cli-lab==0.1.1
azure-cli-monitor==0.2.3
azure-cli-network==2.2.5
azure-cli-nspkg==3.0.3
azure-cli-profile==2.1.1
azure-cli-rdbms==0.3.2
azure-cli-redis==0.3.2
azure-cli-reservations==0.4.0
azure-cli-resource==2.1.4
azure-cli-role==2.1.5
azure-cli-search==0.1.1
azure-cli-servicebus==0.2.3
azure-cli-servicefabric==0.1.3
azure-cli-sf==1.0.8
azure-cli-sql==2.1.4
azure-cli-storage==2.2.2
azure-cli-telemetry==1.0.0
azure-cli-vm==2.2.3
azure-common==1.1.16
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.5
azure-datalake-store==0.0.31
azure-eventgrid==1.2.0
azure-graphrbac==0.40.0
azure-keyvault==1.1.0
azure-loganalytics==0.1.0
azure-mgmt==4.0.0
azure-mgmt-advisor==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.50.0
azure-mgmt-batch==5.0.1
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-cdn==3.0.0
azure-mgmt-cognitiveservices==3.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==4.1.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.1.0
azure-mgmt-containerregistry==2.2.0
azure-mgmt-containerservice==4.2.2
azure-mgmt-cosmosdb==0.4.0
azure-mgmt-datafactory==0.6.0
azure-mgmt-datalake-analytics==0.2.0
azure-mgmt-datalake-nspkg==3.0.0
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==0.1.0
azure-mgmt-devspaces==0.1.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==0.4.0
azure-mgmt-eventhub==2.1.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-iotcentral==0.1.0
azure-mgmt-iothub==0.6.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-loganalytics==0.2.0
azure-mgmt-logic==3.0.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managementgroups==0.1.0
azure-mgmt-managementpartner==0.1.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==1.0.0rc1
azure-mgmt-monitor==0.5.2
azure-mgmt-msi==0.2.0
azure-mgmt-network==2.2.1
azure-mgmt-notificationhubs==2.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.1.0
azure-mgmt-powerbiembedded==2.0.0
azure-mgmt-rdbms==1.3.0
azure-mgmt-recoveryservices==0.1.0
azure-mgmt-recoveryservicesbackup==0.1.1
azure-mgmt-redis==5.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.3.0
azure-mgmt-resource==2.0.0
azure-mgmt-scheduler==2.0.0
azure-mgmt-search==2.0.0
azure-mgmt-servicebus==0.5.1
azure-mgmt-servicefabric==0.2.0
azure-mgmt-signalr==0.1.1
azure-mgmt-sql==0.9.1
azure-mgmt-storage==2.0.0rc4
azure-mgmt-subscription==0.2.0
azure-mgmt-trafficmanager==0.50.0
azure-mgmt-web==0.40.0
azure-multiapi-storage==0.2.2
azure-nspkg==3.0.2
azure-servicebus==0.21.1
azure-servicefabric==5.6.130
azure-servicemanagement-legacy==0.20.6
azure-storage-blob==1.1.0
azure-storage-common==1.1.0
azure-storage-file==1.3.1
azure-storage-nspkg==3.0.0
azure-storage-queue==1.3.0
msrest==0.5.5
msrestazure==0.5.0
@nicholasmhughes

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2018

so, the API version that the 2018.3 azurearm driver was written against is extremely old. there's likely a mismatch between the API version that your SDK wants to use and the one that the driver expects (since it's not pinned).

you could try to use the azurearm code in the develop branch (which I think made it into Fluorine) and see if that resolves the issues. however, I'd expect that you'll still run into problems at the SDK levels you're running. I can dig up some SDK versions that should be good if you're willing to downgrade them and run the develop code...

@kiemlicz

This comment has been minimized.

Copy link
Contributor Author

commented Oct 3, 2018

Yes, I'll check the develop branch but you mean I should downgrade azure python libs (if you could tell me to which version that would be very helpful)?
But...
Basically after the next release it is possible that it's still broken?
Why this azurearm cloud module made it to Salt then?
Do I have any options to spawn some resources on Azure (besides develop branch)?

@nicholasmhughes

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2018

Correct, there's a delicate balance in which Azure Python SDK versions can be used. To be clear, this is due to the fact that the azurearm driver is not currently pinned to a specific API version. The SDK will attempt to use whatever it has as its "latest" API version unless you tell it otherwise.

This code "made it to Salt" because it did work at one time. Microsoft has been making major changes at a furious pace. Without pinning the API version, you can only try to accommodate model changes in the API by dynamically generating the objects that the azurearm driver uses (which I contributed in the develop code). However, this doesn't fix the fact that MS often decides to introduce required parameters in later API versions which weren't previously present. This puts us in a spot where we're able to build the object, but it isn't validated properly... and explodes.

I think your best bet is to grab the azurearm cloud module and util module from develop and shoot them into your _clouds and _utils directories respectively in the salt filesystem root. Don't forget to salt-run saltutil.sync_all. This will also give you important things like managed disks, which aren't present in the 2018.3 code. From there it's just a matter of figuring out what SDK version you need. I won't be able to get you my pip freeze output for a bit, but I have some comments on issues out there where I talk about specific versions that worked, like the one here.

I know it's frustrating to get set up, but it's worth it once you get it rolling. I think the main problem is that the cloud stuff is moving faster than the Salt releases. I contributed most of the code in develop at the beginning of this year. I think the Azure Python SDK has jumped two major versions since then. Feel free to tag me on any issues that pop up while you're trying to get this working.

@nicholasmhughes

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2018

Here's what I'm running right now:

azure==3.0.0
azure-batch==4.1.3
azure-common==1.1.10
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.2
azure-datalake-store==0.0.19
azure-eventgrid==0.1.0
azure-graphrbac==0.40.0
azure-keyvault==0.3.7
azure-mgmt==2.0.0
azure-mgmt-advisor==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.30.0
azure-mgmt-batch==5.0.0
azure-mgmt-batchai==0.2.0
azure-mgmt-billing==0.1.0
azure-mgmt-cdn==2.0.0
azure-mgmt-cognitiveservices==2.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==3.1.0rc3
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==0.3.1
azure-mgmt-containerregistry==1.0.1
azure-mgmt-containerservice==3.0.1
azure-mgmt-cosmosdb==0.3.1
azure-mgmt-datafactory==0.4.0
azure-mgmt-datalake-analytics==0.3.0
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.3.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==1.2.0
azure-mgmt-documentdb==0.1.3
azure-mgmt-eventgrid==0.4.0
azure-mgmt-eventhub==1.2.0
azure-mgmt-hanaonazure==0.1.0
azure-mgmt-iothub==0.4.0
azure-mgmt-iothubprovisioningservices==0.1.0
azure-mgmt-keyvault==0.40.0
azure-mgmt-loganalytics==0.1.0
azure-mgmt-logic==2.1.0
azure-mgmt-machinelearningcompute==0.4.0
azure-mgmt-managementpartner==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==0.2.0
azure-mgmt-monitor==0.4.0
azure-mgmt-msi==0.1.0
azure-mgmt-network==2.0.0rc1
azure-mgmt-notificationhubs==1.0.0
azure-mgmt-nspkg==2.0.0
azure-mgmt-powerbiembedded==1.0.0
azure-mgmt-rdbms==0.1.0
azure-mgmt-recoveryservices==0.2.0
azure-mgmt-recoveryservicesbackup==0.1.1
azure-mgmt-redis==5.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.1.0
azure-mgmt-resource==1.2.2
azure-mgmt-scheduler==1.1.3
azure-mgmt-search==1.0.0
azure-mgmt-servermanager==1.2.0
azure-mgmt-servicebus==0.4.0
azure-mgmt-servicefabric==0.1.0
azure-mgmt-sql==0.8.6
azure-mgmt-storage==1.5.0
azure-mgmt-subscription==0.1.0
azure-mgmt-trafficmanager==0.40.0
azure-mgmt-web==0.34.1
azure-monitor==0.3.1
azure-nspkg==2.0.0
azure-servicebus==0.21.1
azure-servicefabric==6.1.2.9
azure-servicemanagement-legacy==0.20.6
azure-storage==0.36.0
azure-storage-blob==1.1.0
azure-storage-common==1.1.0
azure-storage-file==1.1.0
azure-storage-nspkg==3.0.0
azure-storage-queue==1.1.0
msrest==0.4.25
msrestazure==0.4.25
@kiemlicz

This comment has been minimized.

Copy link
Contributor Author

commented Oct 11, 2018

Thank you
I'm fine with taking the cloud modules from develop branch
I've upgraded to the very same versions as yours and:

  • I'm able to list the sizes, images etc.
  • I can issue VM creation, however:
    -- some VM sizes are not available even though they were listed with list-sizes, I think this is due to still old library version, am I right?
    -- ssh_password is do required contrary to what the doc says: ssh_publickeyfile Use either ssh_publickeyfile or ssh_password. ...

However VM creation either fails with:

  1. /etc/salt/cloud.profiles.d/myprofile:
azure-ubuntu-small:
  provider: edited
  image: 'Canonical|UbuntuServer|18.10-DAILY|18.10.201810100'
  size: Standard_D2_v2
  location: edited
  ssh_username: edited
  ssh_password: edited
  ssh_keyfile: "/etc/salt/pki/master/ssh/salt-ssh.rsa"
  ssh_publickeyfile: "/etc/salt/pki/master/ssh/salt-ssh.rsa.pub"
  resource_group: edited
  network: edited
  subnet: default

I receive

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/cli.py", line 281, in run
    self.config.get('names')
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1454, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1284, in create
    output = self.clouds[func](vm_)
  File "/var/cache/salt/master/extmods/clouds/azurearm.py", line 1450, in create
    'wait_for_ip_interval_multiplier', vm_, __opts__, default=1),
  File "/usr/lib/python2.7/dist-packages/salt/utils/cloud.py", line 2389, in wait_for_ip
    data = update_callback(*update_args, **update_kwargs)
  File "/var/cache/salt/master/extmods/clouds/azurearm.py", line 1434, in _query_node_data
    ip_address = data['public_ips'][0]
IndexError: list index out of range

  1. (After looking at #44886 (comment)) Changing: /etc/salt/cloud.profiles.d/myprofile:
azure-ubuntu-small:
  provider: edited
  image: 'Canonical|UbuntuServer|18.10-DAILY|18.10.201810100'
  size: Standard_D2_v2
  location: edited
  ssh_username: edited
  ssh_password: edited
  resource_group: edited
  network: edited
  subnet: default
  allocate_public_ip: False
  bootstrap_interface: private

Works but without the public IP allocation which is far too limiting.

  1. Changing: /etc/salt/cloud.profiles.d/myprofile:
azure-ubuntu-small:
  provider: edited
  image: 'Canonical|UbuntuServer|18.10-DAILY|18.10.201810100'
  size: Standard_D2_v2
  location: edited
  ssh_username: edited
  ssh_password: edited
  resource_group: edited
  network: edited
  subnet: default
  allocate_public_ip: True

yields:

[ERROR   ] There was a profile error: __init__() takes exactly 1 argument (2 given)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/cli.py", line 281, in run
    self.config.get('names')
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1454, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1284, in create
    output = self.clouds[func](vm_)
  File "/var/cache/salt/master/extmods/clouds/azurearm.py", line 1418, in create
    vm_request = request_instance(vm_=vm_)
  File "/var/cache/salt/master/extmods/clouds/azurearm.py", line 1013, in request_instance
    kwargs=vm_
  File "/var/cache/salt/master/extmods/clouds/azurearm.py", line 859, in create_network_interface
    six.text_type(pub_ip_data.id),  # pylint: disable=no-member
TypeError: __init__() takes exactly 1 argument (2 given)
@kiemlicz

This comment has been minimized.

Copy link
Contributor Author

commented Oct 11, 2018

ad 3. After adding following change:
azurearm.py:857
from:

ip_kwargs['public_ip_address'] = PublicIPAddress(
     six.text_type(pub_ip_data.id),  # pylint: disable=no-member
)

to

ip_kwargs['public_ip_address'] = PublicIPAddress(
    id=six.text_type(pub_ip_data.id),  # pylint: disable=no-member
)

The VM creation works
I guess this is bug?

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2018

I wonder if they changed the order of args in azurearm... wouldn't surprise me.

If you wanna open that as a PR, that would be great!

kiemlicz added a commit to kiemlicz/salt that referenced this issue Oct 11, 2018

https://github.com/saltstack/issues/49883
Using named argument in PublicIPAddress azure network_models API call

gtmanfred added a commit that referenced this issue Oct 12, 2018

@nicholasmhughes

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2018

@kiemlicz to answer your remaining questions:

  • yes, sizes are codified in SDK... so new sizes might not be available without using the newer SDK versions.
  • it's curious that you had to provide a ssh_password... I'll have to test that out when I get a chance, because that definitely wasn't the case before...

gitebra pushed a commit to gitebra/salt that referenced this issue Oct 15, 2018

Merge remote-tracking branch 'upstream/develop' into develop
* upstream/develop: (35 commits)
  Make sure from-filenames intersect with names-file
  Expose docs for Ansible modules
  Unit Tests for RosterMatcher
  docs: Correct spelling mistake and smooth out sentence
  runtests.py: Accept modified file list from a text file
  enable testing only filemaps
  Fix test errors
  saltstack#49883
  Use `from ... import` for the salt exceptions, as recommended Fixed the integration test coding style error Directly use the task module instead of using run_function. We didn't need the extra feature it provided
  pylint error
  pylint errors
  Fix virt.init documentation
  Turn virt.pool_info name parameter optional
  Turn virt.network_info name parameter optional
  Fix pylint errors in virt module
  Increase centos7-py2 kitchen pr timeout to 8 hours
  Corrected the win_task.create_task_from_xml test to fit with the new API.
  Raise a CommandExecutionError instead of returning the error message, as suggested in the core review.
  Added an integration test for win_task.create_task_from_xml
  Changed the documentation to reflect the change. win_task.create_task_from_xml can now return a string when there was an error Fixed a bug where win_task.create_task_from_xml did not worked when the username was set to None
  ...

rallytime added a commit to rallytime/salt that referenced this issue Oct 15, 2018

https://github.com/saltstack/issues/49883
Using named argument in PublicIPAddress azure network_models API call
@ol3k

This comment has been minimized.

Copy link

commented Oct 31, 2018

I was able to set up a working salt-cloud module because of this thread.

I think your best bet is to grab the azurearm cloud module and util module from develop and shoot them into your _clouds and _utils directories respectively in the salt filesystem root. Don't forget to salt-run saltutil.sync_all. This will also give you important things like managed disks, which aren't present in the 2018.3 code. From there it's just a matter of figuring out what SDK version you need. I won't be able to get you my pip freeze output for a bit, but I have some comments on issues out there where I talk about specific versions that worked, like the one here.

This should be definitely mentioned in any doc or pushed to top.
It's just annoying, a game of try&error and costs at least hours of debugging which modules/packages/config you have to use in order to get salt-cloud running in Oct 2018.

The actual dev release (as by 31/10) seems to work together with the docs for azurearm. The pip freeze output above gives good hints which pip packages to use.

I know that's not a problem of salt / msazure in general but perhaps someone finds this thread and gets faster through the different (sometimes misleading) errors occurring during the deploy tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.