Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Salt-cloud - proxmox driver - returns AttributeError: 'generator' object has no attribute 'next' #49559

Open
zyguy opened this issue Sep 7, 2018 · 2 comments

Comments

@zyguy
Copy link

commented Sep 7, 2018

Description of Issue/Question

My goal is to create lxc container from salt-cloud with the proxmox driver.

When trying to create lxc container on a proxmox node, the salt-cloud command fails with this error :

The following exception was thrown when trying to run the initial deployment:
'generator' object has no attribute 'next'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 541, in create
    data = create_node(vm_, newid)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 727, in create_node
    static_props):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 652, in _get_properties
    sub = (item for item in sub if item["path"] == search_path).next()['children']
AttributeError: 'generator' object has no attribute 'next'
Error: There was a profile error: Failed to deploy VM

In the pveproxy/acces.log I can see the salt-cloud requests :

192.168.0.44 - - [07/09/2018:15:42:38 +0200] "POST /api2/json/access/ticket HTTP/1.1" 200 656
192.168.0.44 - root@pam [07/09/2018:15:42:38 +0200] "GET /api2/json/cluster/resources HTTP/1.1" 200 752
192.168.0.44 - root@pam [07/09/2018:15:42:38 +0200] "GET /api2/json/nodes/proxmox/lxc/101/config HTTP/1.1" 200 259
192.168.0.44 - - [07/09/2018:15:42:42 +0200] "POST /api2/json/access/ticket HTTP/1.1" 200 661
192.168.0.44 - root@pam [07/09/2018:15:42:42 +0200] "GET /api2/json/cluster/nextid HTTP/1.1" 200 14
192.168.0.44 - - [07/09/2018:15:42:42 +0200] "GET /pve-docs/api-viewer/apidoc.js HTTP/1.1" 200 107671

According to the pve-api the endpoint should be /api2/json/nodes/{node}/lxc but I get a HTTP 500 return code when I manually send POST to it.

192.168.0.32 - root@pam [07/09/2018:15:47:06 +0200] "POST /api2/json/nodes/proxmox/lxc HTTP/1.1" 500 13

I check the pveproxy/access.log when I create a lxc container from the proxmox webui and saw a different endpoint used :

192.168.0.32 - root@pam [07/09/2018:15:24:39 +0200] "POST /api2/extjs/nodes/proxmox/lxc HTTP/1.1" 200 85

I can create a lxc container with the pve-api using the endpoint /api2/extjs/nodes/proxmox/lxc with the following data :

def create_lxc(token, ticket):
  endpoint = "/api2/extjs/nodes/proxmox/lxc"
  url = host+endpoint
  payload = "vmid=103&hostname=lxc103&password=password&ostemplate=local%3Avztmpl%2Fdebian-9.0-standard_9.5-1_amd64.tar.gz&rootfs=local-lvm%3A1&cores=1&memory=512&swap=512&net0=bridge%3Dvmbr0%2Cname%3Deth0"
  headers = { 
      'content-type': "application/x-www-form-urlencoded",
      "CSRFPreventionToken" : token,
      }  
  cookie = {'PVEAuthCookie': ticket}
  response = requests.request("POST", url, data=payload, headers=headers, cookies=cookie, verify=False)  
  return response.json()  

The code return {'success': 1, 'data': 'UPID:proxmox:000020EC:000599AB:5B928396:vzcreate:103:root@pam:'}

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

#/etc/salt/cloud.providers.d/proxmox.conf
pxmx1:
  minion:
    master: salt.local
  user: user@pam
  password: password
  url: proxmox.local
  driver: proxmox
  verify_ssl: False

#/etc/salt/cloud.profiles.d/lxc.conf
lxc:
  provider: pxmx1
  image: local:vztmpl/debian-8.0-standard_8.7-1_amd64.tar.gz
  technology: lxc
  ip_address: ip.of.pxmx1
  password: password

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

root@salt:[/srv/cloud/salt-cloud.config]: salt-cloud -p lxc lxc102 -l debug
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/proxmox.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/proxmox.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/lxc.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/lxc.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] Could not LazyLoad proxmox.avail_sizes: 'proxmox.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad proxmox.optimize_providers: 'proxmox.optimize_providers' is not available.
[DEBUG   ] The 'proxmox' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad proxmox.list_nodes_min: 'proxmox.list_nodes_min' is not available.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] Could not LazyLoad proxmox.avail_sizes: 'proxmox.avail_sizes' is not available.
[DEBUG   ] Getting resource: vms.. (filter: None)
[DEBUG   ] Not authenticated yet, doing that now..
[WARNING ] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)

[DEBUG   ] get: https://proxmox.local:8006/api2/json/cluster/resources (None)
[DEBUG   ] get: https://proxmox.local:8006/api2/json/nodes/proxmox/lxc/101/config (None)
[DEBUG   ] Filter not given: {'lxc101': {'name': 'lxc101', 'maxdisk': 4294967296, 'maxmem': 536870912, 'node': 'proxmox', 'mem': 0, 'template': 0, 'disk': 0, 'type': 'lxc', 'uptime': 0, 'maxcpu': 1, 'vmid': 101, 'diskwrite': 0, 'status': 'stopped', 'netout': 0, 'id': 'lxc/101', 'config': {'rootfs': 'local-lvm:vm-101-disk-1,size=4G', 'ostype': 'debian', 'swap': 512, 'hostname': 'lxc101', 'arch': 'amd64', 'net0': 'name=eth0,bridge=vmbr0,hwaddr=AE:24:8D:00:59:53,type=veth', 'digest': '60c0617e5ef3607e09c8ee17c1049b98afec857c', 'memory': 512, 'cores': 1}, 'diskread': 0, 'cpu': 0, 'netin': 0}}, returning all resource: nodes
[DEBUG   ] VM_Name: lxc101
[DEBUG   ] vm_details: {'name': 'lxc101', 'maxdisk': 4294967296, 'maxmem': 536870912, 'node': 'proxmox', 'mem': 0, 'template': 0, 'disk': 0, 'type': 'lxc', 'uptime': 0, 'maxcpu': 1, 'vmid': 101, 'diskwrite': 0, 'status': 'stopped', 'netout': 0, 'id': 'lxc/101', 'config': {'rootfs': 'local-lvm:vm-101-disk-1,size=4G', 'ostype': 'debian', 'swap': 512, 'hostname': 'lxc101', 'arch': 'amd64', 'net0': 'name=eth0,bridge=vmbr0,hwaddr=AE:24:8D:00:59:53,type=veth', 'digest': '60c0617e5ef3607e09c8ee17c1049b98afec857c', 'memory': 512, 'cores': 1}, 'diskread': 0, 'cpu': 0, 'netin': 0}
[DEBUG   ] Generating minion keys for 'lxc102'
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/lxc102/creating; data = {'name': 'lxc102', 'profile': 'lxc', '_stamp': '2018-09-07T13:34:20.832944', 'driver': 'proxmox', 'event': 'starting create', 'provider': 'pxmx1:proxmox'}
[INFO    ] Creating Cloud VM lxc102
[DEBUG   ] Not authenticated yet, doing that now..
[WARNING ] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)

[DEBUG   ] get: https://proxmox.local:8006/api2/json/cluster/nextid (None)
[ERROR   ] Error creating lxc102 on PROXMOX

The following exception was thrown when trying to run the initial deployment:
'generator' object has no attribute 'next'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 541, in create
    data = create_node(vm_, newid)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 727, in create_node
    static_props):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/proxmox.py", line 652, in _get_properties
    sub = (item for item in sub if item["path"] == search_path).next()['children']
AttributeError: 'generator' object has no attribute 'next'
Error: There was a profile error: Failed to deploy VM

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 (default, Nov 23 2017, 16:37:01)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-134-generic
         system: Linux
        version: Ubuntu 16.04 xenial

root@salt:[/srv/cloud/salt-cloud.config]: dpkg -l|grep -E 'requests|IPy'
ii  python-ipy                          1:0.83-1                                   all          Python module for handling IPv4 and IPv6 addresses and networks
ii  python-requests                     2.9.1-3                                    all          elegant and simple HTTP library for Python2, built for human beings
ii  python3-ipy                         1:0.83-1                                   all          Python3 module for handling IPv4 and IPv6 addresses and networks
ii  python3-requests                    2.9.1-3                                    all          elegant and simple HTTP library for Python3, built for human beings

root@salt:[/srv/cloud/salt-cloud.config]: pip3 list|grep -E 'requests|IPy'
IPy (0.83)
requests (2.9.1)

root@proxmox:[~]: pveversion
pve-manager/5.2-1/0fcd7879 (running kernel: 4.15.17-1-pve)
@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@zyguy Thanks for the report. To clarify, if you update the API URL in Salt to the one used by the webui then everything functions as expected? I wonder if this was a change in Proxmox?

@zyguy

This comment has been minimized.

Copy link
Author

commented Sep 12, 2018

@garethgreenaway Thank's for your time. I think that the other endpoint is just a renaming from the js webui and the final one is the good one used by the promox driver.

After seeking into the code why I raised this error and comment some part of the validation of _get_properties() to bypass the errors, i found that it should be around the lines 654 with the next() method use to return the parameter list from api for defined path and HTTP method.

I couldn't get the right properties and the POST data isn't good enough for pve-api.

In fact the exactly same config run well with salt-master-py2 and not well with salt-master-py3.

root@saltpy2:[/etc/salt]: cat cloud.providers.d/promox.conf
pxmx1:
  minion:
    master: salt.local

  user: root@pam
  password: password

  url: proxmox.local
  driver: proxmox
  verify_ssl: False

root@saltpy2:[/etc/salt]: cat cloud.profiles.d/lxc.conf
lxc:
  provider: pxmx1
  image: local:vztmpl/debian-9.0-standard_9.5-1_amd64.tar.gz
  technology: lxc
  # host needs to be set to the configured name of the proxmox host
  # and not the ip address or FQDN of the server
  host: proxmox # name visible from the cluster webadmin here
  ip_address: 192.168.0.47
  password: proxmox
  onboot: 1
  storage: local-lvm
#  rootfs: volume=local-lvm:4GB
  cores: 1
  memory: 256
  swap: 64
  net0: bridge=vmbr0,name=eth0,ip=9.8.7.6/30
salt-master-py2   :  Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
salt-master-py3   :  Python: 3.5.2 (default, Nov 23 2017, 16:37:01)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.