salt-cloud deployment fails when deploy: True

[sebw]

I'm deploying from a RedHat Enterprise 7 template where salt-minion 2016.3.1 is already installed (from official repositories) with a basic configuration. For different reasons I can't use the bootstrap script.

I use a custom script to finish the deployment, but I get a traceback and the minion key is rejected from the Master. Salt-minion refuses to start as a result. I'm getting the same error using bootstrap-salt.

Accepted Keys:
stestswa32dev.intranet.example.org
Denied Keys:
stestswa32dev.intranet.example.org

salt-minion -l debug:

[CRITICAL] The Salt Master has rejected this minion's public key!
To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

In the template, Salt-minion is set not to start at boot (or I get noname as minion_id). The custom script /etc/salt/cloud.deploy.d/vmware-finishing.sh is supposed to enable salt-minion and start it.

#!/bin/bash

/usr/sbin/chkconfig salt-minion on
/usr/sbin/service salt-minion start

exit 0

This is the profile:

rhel7-DEV-small:
  provider: vmware
  clonefrom: TMPL-RHEL7
  datastore: LDLI01SASHA
  num_cpu: 2
  memory: 4GB
  folder: SALTCLOUD
  datacenter: LEO2
  cluster: Mixed
  host: sesx63.intranet.example.org
  ssh_username: root
  password: xxx
  devices:
    network:
      Network adapter 1:
        name: VLAN401
        adapter_type: vmxnet3
        ip: 172.24.1.82
        subnet_mask: 255.255.255.0
        gateway: 172.24.1.1
        switch_type: standard
  domain: intranet.example.org
  dns_servers:
    - 193.x.x.x
    - 193.x.x.x
  deploy: True
  script: /etc/salt/cloud.deploy.d/vmware-finishing.sh

This is the stacktrace:

[ERROR   ] There was a profile error: ''
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 284, in run
    self.config.get('names')
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1446, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1281, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2442, in create
    data = show_instance(vm_name, call='action')
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 1576, in show_instance
    vm_list = salt.utils.vmware.get_mors_with_properties(_get_si(), vim.VirtualMachine, vm_properties)
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 445, in get_mors_with_properties
    content = get_content(service_instance, object_type, property_list=property_list, container_ref=container_ref)
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 351, in get_content
    container_ref = service_instance.content.rootFolder
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 574, in __call__
    return self.f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor
    return self._stub.InvokeAccessor(self, info)
  File "/usr/lib/python2.7/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
    options=self._pcType.RetrieveOptions(maxObjects=1))
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1327, in InvokeMethod
    resp = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1051, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 379, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

Master:

Salt Version:
            Salt: 2016.3.1

Dependency Versions:
 Apache Libcloud: 0.20.1
            cffi: Not Installed
        cherrypy: 3.2.2
        dateutil: 1.5
           gitdb: Not Installed
       gitpython: Not Installed
           ioflo: Not Installed
          Jinja2: 2.7.2
         libgit2: Not Installed
         libnacl: Not Installed
        M2Crypto: 0.21.1
            Mako: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.4.7
    mysql-python: Not Installed
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
          Python: 2.7.5 (default, Oct 11 2015, 17:47:16)
    python-gnupg: Not Installed
          PyYAML: 3.10
           PyZMQ: 14.7.0
            RAET: Not Installed
           smmap: Not Installed
         timelib: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.5

System Versions:
            dist: redhat 7.2 Maipo
         machine: x86_64
         release: 3.10.0-327.el7.x86_64
          system: Linux

Minion:

[root@stestswa32dev ~]# salt-minion --versions
Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 11 2015, 17:47:16)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: redhat 7.2 Maipo
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.2 Maipo

[Ch3LL]

@sebw what versions of pvmomi and httplib are you using?

Also to be clear this only occurs when using a custom script correct?

[sebw]

@Ch3LL yes it happens with any script actually, not only custom.

pyvmomi (6.0.0.2016.6)

Regarding httplib, this is what I find:

# rpm -qf /usr/lib64/python2.7/httplib.py
python-libs-2.7.5-34.el7.x86_64

This is the latest version of the package available on RHEL7:

# rpm -qi python-libs
Name        : python-libs
Version     : 2.7.5
Release     : 34.el7
Architecture: x86_64
Install Date: Mon 30 Nov 2015 04:47:45 PM CET
Group       : Applications/System
Size        : 24675265
License     : Python
Signature   : RSA/SHA256, Wed 21 Oct 2015 09:43:55 PM CEST, Key ID 199e2f91fd431d51
Source RPM  : python-2.7.5-34.el7.src.rpm
Build Date  : Mon 12 Oct 2015 12:19:12 AM CEST
Build Host  : x86-019.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.python.org/
Summary     : Runtime libraries for Python
Description :
This package contains runtime libraries for use by Python:
- the libpython dynamic library, for use by applications that embed Python as
a scripting language, and by the main "python" executable
- the Python standard library

[sebw]

I have a custom grain module polling our CMDB to retrieve information using httplib.
I think this might be the issue, I'm going to investigate and report back.

[sebw]

I disabled my custom grains so they don't interfere and I'm still getting the same trace. I tried to install httplib2 (pip install httplib2) but it doesn't help.

[Ch3LL]

Would you mind attempting to install an older version of pyvmomi pip install pyVmomi==5.5.0.2014.1.1

I just want to rule out that this is not the issue you are running into.

Also you state that you are deploying from a redhat7 template with salt already installed. Have you attempted to build this out from a different OS template and do you still see the same error?

[steverweber]

I have a similar error as @sebw using:

root@salt-cloud-p01:~# salt --version
salt 2016.11.0-187-g8b9648c
python --version
Python 2.7.12
pip install --upgrade pyvmomi
pip install --upgrade httplib2

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1753, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1709, in wrapper
    return f(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/states/cloud_instance.py", line 50, in present
    vm_instance = __salt__['cloud.get_instance'](name)
  File "/usr/lib/python2.7/dist-packages/salt/modules/cloud.py", line 176, in get_instance
    data = action(fun='show_instance', names=[name], provider=provider)
  File "/usr/lib/python2.7/dist-packages/salt/modules/cloud.py", line 237, in action
    info = client.action(fun, cloudmap, names, provider, instance, kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 482, in action
    return mapper.do_action(names, kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1529, in do_action
    vm_name, call='action'
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/vmware.py", line 1719, in show_instance
    vm_list = salt.utils.vmware.get_mors_with_properties(_get_si(), vim.VirtualMachine, vm_properties)
  File "/usr/lib/python2.7/dist-packages/salt/utils/vmware.py", line 738, in get_mors_with_properties
    local_properties=local_properties)
  File "/usr/lib/python2.7/dist-packages/salt/utils/vmware.py", line 622, in get_content
    container_ref = service_instance.content.rootFolder
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 574, in __call__
    return self.f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor
    return self._stub.InvokeAccessor(self, info)
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
    options=self._pcType.RetrieveOptions(maxObjects=1))
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 580, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/local/lib/python2.7/dist-packages/pyVmomi/SoapAdapter.py", line 1331, in InvokeMethod
## NOTE: I added some extra data to the Exception
    raise Exception(self.path, req, headers)
Exception: ('/sdk', '<?xml version="1.0" encoding="UTF-8"?>\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n<soapenv:Body><RetrievePropertiesEx xmlns="urn:vim25"><_this type="PropertyCollector">propertyCollector</_this><specSet><propSet><type>ServiceInstance</type><all>false</all><pathSet>content</pathSet></propSet><objectSet><obj type="ServiceInstance">ServiceInstance</obj><skip>false</skip></objectSet></specSet><options><maxObjects>1</maxObjects></options></RetrievePropertiesEx></soapenv:Body>\n</soapenv:Envelope>', {'SOAPAction': '"urn:vim25/6.0"', 'Content-Type': 'text/xml; charset=UTF-8', 'Cookie': 'vmware_soap_session="???????????"; Path=/; HttpOnly; Secure;', 'Accept-Encoding': 'gzip, deflate'})

---

when switch to using:

pip install pyvmomi==5.5.0-2014.1.1

also has issues.. because the salt vmware driver is using some keywords that are not available.

[DEBUG   ] Failed to execute 'vmware.list_nodes_min()' while querying for running nodes: SmartConnect() got an unexpected keyword argument 'b64token'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2383, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/vmware.py", line 1486, in list_nodes_min
    vm_list = salt.utils.vmware.get_mors_with_properties(_get_si(), vim.VirtualMachine, vm_properties)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/vmware.py", line 263, in _get_si
    port=port)
  File "/usr/lib/python2.7/dist-packages/salt/utils/vmware.py", line 351, in get_service_instance
    domain)
  File "/usr/lib/python2.7/dist-packages/salt/utils/vmware.py", line 209, in _get_service_instance
    mechanism=mechanism)
TypeError: SmartConnect() got an unexpected keyword argument 'b64token'

so i removed them from 3 calls.

vi /usr/lib/python2.7/dist-packages/salt/utils/vmware.py
## around line 206
REOMVE: 
                    b64token=token,
                    mechanism=mechanism)

also since we are using a selfsigned cert needed to
CHANGE sslContext to something like...
        import ssl
        default_context = ssl._create_default_https_context
        ssl._create_default_https_context = ssl._create_unverified_context
        service_instance = SmartConnect(
            host=host,
            user=username,
            pwd=password,
            protocol=protocol,
            port=port
        )
        ssl._create_default_https_context = default_context

then after all that... still hit issues... I forget what, just that some calls didn't work.

[steverweber]

@nmadhok perhaps you have feedback.

i'm going to install a signed cert today... if this does not resolve the issue then going to have to start some low level debugging.

[steverweber]

note posted a hack fix in #36598. please give it a try.

[Ch3LL]

Okay I am able to replicate this but ONLY when using the python api:

client= CloudClient(path='/etc/salt/cloud',config_dir='/etc/salt/cloud.profiles.d')
client.profile('qa-test-vm', names=['megtest3'])

When I run it normally with salt-cloud -p qa-test-vm megtest3 it works just fine. Is this what you guy sare seeing as well? ping @sebw @steverweber

[steverweber]

this will work because it does not use the cached connection.

salt-cloud -p qa-test-vm megtest3

try this code listed in #36598

where I have a state file that makes more than one call using cloud

t1:
    cloudtest.get_instance:
        - name: salt-play-test.xxx.ca
t2:
    cloudtest.get_instance:
        - name: salt-play-test.xxx.ca
...

[steverweber]

@Ch3LL opps i see im already talking to you in the other thread :)

[Ch3LL]

Thanks for clarifying that for me. Just wanted to make sure the original reporter was not just simply running -p when they initially ran into this issue.

[meggiebot]

@sebw can you test the fix in #37639 ? We believe that fixed this.

[sebw]

Will do on Monday and report back. Thanks :)