salt-cloud vsphere cloning and creating vm's

[ingwa]

Hi there,

I've finally managed to get salt cloud talking correctly to esxi and vsphere in general. Turns out to clone images or create vm's on esxi, you MUST have vsphere center server or similar orchestration management system from them in place to expose those elements of the api. The esxi api does not have that in place.

Having issues now on machine creation, seems there is a string int mixup somewhere:

[DEBUG   ] clone_kwargs are set to {'datastore': None,
 'folder': None,
 'host': None,
 'name': 'newlinux',
 'resourcepool': 'resgroup-11',
 'template': False}
[ERROR   ] There was a profile error: string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/cli.py", line 239, in run
    self.config.get('names')
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1364, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1234, in create
    output = self.clouds[func](vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 248, in create
    deploy_kwargs = _deploy(vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 308, in _deploy
    ip_address = wait_for_ip(vm_['name'])
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 293, in wait_for_ip
    log.debug('Pulling VM {0} {1} seconds for an IP address'.format(vm_['name']))
TypeError: string indices must be integers, not str

Here is my profile:

my-ubuntu-vm:
  provider: my-provider
  image: ubuntu
  minion:
    master: 192.168.1.15
  ssh_username: userabc
  ssh_password: passxyz
  script_args: -c /tmp/
  resourcepool: resgroup-11

I called it like this:

salt-cloud -p my-ubuntu-vm newlinux -l debug

Any ideas on what I'm doing wrong?

[steverweber]

looks like I caused this issue. I'll take a look.

[steverweber]

#17071

[steverweber]

to work around this you can set "deploy: False"

        mfcf_centos6:
            provider: mfcf
            image: _centos_6_5
            resourcepool: resgroup-41098
            folder: TEST
            deploy: False

[rallytime]

@steverweber Thanks for the quick PR!

@ingwa Can you give this fix a try when you have a moment? I am also backporting the fix to the 2014.7 branch because it looks like that code was changed there as well.

[rallytime]

Actually, I was looking at the wrong branch. Looks like this bug is only on develop, so I won't be backporting the PR. My mistake!

[ingwa]

@steverweber Hi Steve, I did that and the vm successfully created with a nice output of all the fields. Only thing is salt doesn't appear to have installed.

Versions:

salt-master 2014.7.0-911-ge243a65 (Helium)
salt-minion 2014.7.0-911-ge243a65 (Helium)
salt-cloud 2014.7.0-911-ge243a65 (Helium)

[steverweber]

/ salt is not installed on the new system because deploy: False /

if you reinstall salt using the latest develop branch it will have the fix so you can use deploy: True.

Or you could manualy update your salt code with this fix.
steverweber@f855645

I havent had a chance to test Deploy: True, please let me know if you get it to work.
Things that the documents are likly missing is that VM tools will need to be installed on the template/vm you clone. Also after the new instance boots for the first time, it must have an ip address or get one from dhcp.

[mochrul]

It's not related to the original question, but the vsphere provider does not use the ssh_username and the ssh_password keywords. It uses the template_user(name)? and template_password keywords. At least it not worked for me to use ssh_username. And even there were no default as in any other provider.

[mochrul]

Seeing the code, isn't a patch like this also needed to make it run? I'm really just curious.

diff --git a/salt/cloud/clouds/vsphere.py b/salt/cloud/clouds/vsphere.py
index e3082c2..eb58818 100644
--- a/salt/cloud/clouds/vsphere.py
+++ b/salt/cloud/clouds/vsphere.py
@@ -305,7 +305,7 @@ def deploy(vm):
'''
# TODO: review salt.utils.cloud.bootstrap(vm_, opts)
# TODO: review salt.utils.cloud.wait_for_ip

• ip_address = wait_for_ip(vm_['name'])

• ip_address = wait_for_ip(vm_)

 template_user = config.get_cloud_config_value(
     'template_user', vm_, __opts__

[steverweber]

Thanks, I missed that.

[ingwa]

Hi guys, appreciate all your input!

@steverweber thanks for making the updates, seems we're on the way there. My latest version is:

salt-cloud 2014.7.0-990-g023e835 (Helium)
salt-master 2014.7.0-990-g023e835 (Helium)
salt-minion 2014.7.0-990-g023e835 (Helium)

@mochrul thanks for your input re the passwords.

I now have this in my ubuntu.conf profile:

salt-cloud 2014.7.0-990-g023e835 (Helium)
salt-master 2014.7.0-990-g023e835 (Helium)
salt-minion 2014.7.0-990-g023e835 (Helium)
root@salt:/etc/salt/cloud.profiles.d# vim ubuntu.conf 

ubuntu:
  provider: my-provider
  image: ubuntu
  template_username: "redacted"
  template_password: "redacted"
  ssh_username: "redacted"
  ssh_password: "redacted"
  script_args: -c /tmp/
  resourcepool: resgroup-11
  deploy: True  

When I run the create function, it creates successfully, machine spins up and then it tries to connect. I'm getting this:

[DEBUG   ] Deploying 192.168.1.22 at 1415140802.0
[DEBUG   ] Attempting connection to host 192.168.1.22 on port 22
[DEBUG   ] SSH port 22 on 192.168.1.22 is available
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as None (try 1 of 15)
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 None@192.168.1.22 date'
[DEBUG   ] Child Forked! PID: 25399  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 None@192.168.1.22 date
Warning: Permanently added '192.168.1.22' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.22' (ECDSA) to the list of known hosts.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied, please try again.
[DEBUG   ] Permission denied, please try again.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied, please try again.
[DEBUG   ] Permission denied, please try again.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied (publickey,password).
[DEBUG   ] Permission denied (publickey,password).
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as None (try 2 of 15)

*** Truncated***

Anything I've done?

(corrected spelling in config)

[ingwa]

FYI I have also tried with just user: and password: in the config however the connection debug always states it's using "None".

[ingwa]

Ok, I did some more testing and it seems the template_user: is the correct format for the username, however for some reason the password won't get taken. Here's the output from the debug:

[DEBUG   ] Deploying 192.168.1.4 at 1415142853.0
[DEBUG   ] Attempting connection to host 192.168.1.4 on port 22
[DEBUG   ] SSH port 22 on 192.168.1.4 is available
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as redacteduser (try 1 of 15)
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 date'
[DEBUG   ] Child Forked! PID: 26509  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 date
Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
redacteduser@192.168.1.4's password: [DEBUG   ] redacteduser@192.168.1.4's password:

Tue Nov  4 15:14:50 PST 2014
[DEBUG   ] Tue Nov  4 15:14:50 PST 2014
[DEBUG   ] Logging into 192.168.1.4:22 as redacteduser
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 \'test -e \\"/tmp/.saltcloud-1f679a23-3da4-4abd-a559-3d7673c161eb\\"\''
[DEBUG   ] Child Forked! PID: 26511  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 'test -e \"/tmp/.saltcloud-1f679a23-3da4-4abd-a559-3d7673c161eb\"'
Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
redacteduser@192.168.1.4's password: [DEBUG   ] redacteduser@192.168.1.4's password:
Usage: salt-cloud

salt-cloud: error: There was a profile error: A wrong password has been issued while establishing ssh session

I have successfully logged into the ssh client after the machine spun up manually using the same credentials in the input file. I trimmed it up too by removing ssh_username, etc.

[steverweber]

thanks for testing this. I think I'll take a look tomorrow.

[steverweber]

looking at the source... this wont work... the code will need to be changed :(

I might take the time to migrate the code to use:
salt.utils.cloud.bootstrap

or pass 'has_ssh_agent' to the deploy method so the password will take...

Note to self:
salt / salt / utils / cloud.py

    has_ssh_agent = False
    if opts.get('ssh_agent', False) and 'SSH_AUTH_SOCK' in os.environ:
        if stat.S_ISSOCK(os.stat(os.environ['SSH_AUTH_SOCK']).st_mode):
            has_ssh_agent = True

912: def deploy_script(host,
...
            elif password and 'has_ssh_agent' in kwargs and kwargs['has_ssh_agent'] is False:
                log.debug('Using {0} as the password'.format(password))
                ssh_kwargs['password'] = password

[ingwa]

@steverweber appreciate you looking into this. Once it's all up and running I'll try work on a detailed example for the documentation.

[steverweber]

sounds good. I'll clean up the code a little so it can take in:

ssh_username
ssh_password

I'm tempted to deprecat
http://docs.saltstack.com/en/latest/topics/development/deprecations.html

template_user
template_password

[techhat]

I would support deprecating template_user and template_password.

[ingwa]

Something else that I observed yesterday with all my tests. I notice that part of the command to create a new vm is to define it's name, salt-cloud -p profilename newname. Newname is then used to create a pre-authorized key for salt-key, however when the template vm spins up it's host name is generic, and salt then registers the old hostname from the vm, and places it in unapproved. Secondly, even though I defined:

minion:
  master: mymasterip

I tried setting this in the profile, provider, and cloud configs however nothing would allow overriding the default "salt" local network name.

My next bet is to perhaps write my own startup sh script that pulls the bootstrap, then does a rename on the hostname to get the new name for the vm and use that instead, and then restart salt, but I am wondering isn't this done somehow already?

[ingwa]

FYI, the tests I performed were after applying the hot fix directly to my salt code and running it again. It did work with the overrides in place, but really appreciate you all looking into this to get it working. I have a few cloud systems I'm testing but one step at a time hehe :)

[rallytime]

@steverweber I see you have a commit referencing this issue, but it doesn't appear to be related to a pull request. Are you still working on that, or did it just get missed?

[steverweber]

Sorry, Its been one of them weeks... Had no time. I can say the second week
of December I'll have resolve this.
On Nov 18, 2014 6:26 PM, "Nicole Thomas" notifications@github.com wrote:

@steverweber https://github.com/steverweber I see you have a commit
referencing this issue, but it doesn't appear to be related to a pull
request. Are you still working on that, or did it just get missed?

—
Reply to this email directly or view it on GitHub
#17061 (comment).

[rallytime]

Not a problem at all @steverweber! I just wanted to check in. Thanks again for all of your contributions!

[rallytime]

The vsphere driver is on a deprecation path and will be officially removed in Salt Carbon (see #24487 and #24504). As such, we'd recommend moving to the new vmware driver. Since we have a much more robust and clean driver, we won't be spending time cleaning up bugs for vsphere. However, we will of course be accepting fixes from community members as they are submitted.

[nmadhok]

@rallytime I think this issue can be closed now.

[steverweber]

cheers!

[molixrawdi]

Hi,
Thanks for all your efforts, may I just confirm that users of SaltStack can expect to:
1- Create a virtual machine via template file, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.
2- Clone an existing virtual machine, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.

Best Regards
M

[molixrawdi]

Could you please confirm that this link is deprecated: https://docs.saltstack.com/en/latest/ref/clouds/all/salt.cloud.clouds.vsphere.html

[rallytime]

@molixrawdi Correct - the vsphere driver is officially on a deprecation path and will be removed from Salt in the Carbon release. The VMWare driver in Salt-Cloud should be used instead:

• https://docs.saltstack.com/en/latest/topics/cloud/vmware.html
• https://docs.saltstack.com/en/latest/ref/clouds/all/salt.cloud.clouds.vmware.html

[molixrawdi]

Hi @rallytime many thanks for the update, Am I correct about the two points I made earlier? (1-Create Virtual machine, 2- Clone virtual machine)? By mentioning the vmware drivers as replacement, Am I correct in assuming by this you mean 'pyVmomi' = https://pypi.python.org/pypi/pyvmomi

Best Regards
Molix

[rallytime]

@molixrawdi - There is a salt-cloud driver named vmware.py in the salt/cloud/clouds/ directory of Salt, which is the salt-cloud driver I was referring to above. The docs that I linked in my earlier comment explain how to use that driver. The salt-cloud vmware.py driver depends on the pyVmomi package installable through pip.

Here's the salt-cloud driver I am referring to: https://github.com/saltstack/salt/blob/2015.8/salt/cloud/clouds/vmware.py

Does that help clear things up?

[molixrawdi]

@rallytime Thanks again, so the vmware.py comes with salt-cloud which I have so far managed to install using pip install salt-cloud? (Sorry for asking many questions and I truly appreciate your help and time), finally about my two assumptions are they correct!

They are:

1- Create a virtual machine via template file, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.
2- Clone an existing virtual machine, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.

[rallytime]

@molixrawdi Yes, the vmware.py driver comes with salt-cloud. That driver is how salt-cloud interacts with/connects to your VMWare installation. How did you install salt? Depending on your operating system, salt-cloud is either included by default with your salt installation, or it is an extra package taht you can install. You could use pip as well if you want, but it might already be available. Also, what version of salt are you running? (Please post the output of salt-cloud --versions-report.) This will determine if you have the correct vmware.py driver available by default in your salt installation.

I guess I don't understand what you're stating with your assumptions. You can create and clone a virtual machine with a template file with the vmware driver that comes with salt-cloud. But before you do that you need to install pyVmomi, because the salt-cloud vmware.py driver uses the pyVmomi driver.

From the the top of the vmware driver docs: "The VMware cloud module allows you to manage VMware ESX, ESXi, and vCenter."

[molixrawdi]

Hi @rallytime
I have run the command, please see this output below:

salt01 ~]# salt-cloud --versions-report
Salt Version:
            Salt: 2015.8.0

Dependency Versions:
 Apache Libcloud: 0.18.0
          Jinja2: 2.2.1
        M2Crypto: Not Installed
            Mako: Not Installed
          PyYAML: 3.11
           PyZMQ: 14.5.0
          Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
            RAET: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.5
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 1.4.1
           gitdb: Not Installed
       gitpython: Not Installed
           ioflo: Not Installed
         libnacl: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.4.6
    mysql-python: Not Installed
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
    python-gnupg: Not Installed
           smmap: Not Installed
         timelib: Not Installed

System Versions:
            dist: centos 6.7 Final
         machine: x86_64
         release: 2.6.32-573.7.1.el6.x86_64
          system: CentOS 6.7 Final

[molixrawdi]

I have also a second node for testing running Ubuntu 14.04:

salt02:~$ salt-cloud --versions-report
                  Salt: 2015.5.3
                Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: Not Installed
               Tornado: Not Installed
       Apache Libcloud: 0.18.0
 Debian source package: 2015.5.3+ds-1trusty1

[molixrawdi]

I have tested both with multiple commands from salt to ensure that there are no outstanding compatibility issues with libraries or clashes with versions.

[molixrawdi]

After testing, I came to the conclusion that one must install 'salt-cloud' via 'apt-get' and the salt repository instead of 'pip install' the pip install will fail due to unsolved dependencies.

[rallytime]

@molixrawdi Ok - So the vmware.py driver isn't included by default in the 2015.5.3 release as it was added later to the 2015.5 branch. (It's there in the 2015.8.0 release though.) If you want to use the vmware.py driver in your 2015.5.3 install, you need to grab the driver and put it in your clouds directory. Since you're on Ubuntu, this should look something like /usr/lib/python2.7/dist-packages/salt/cloud/clouds/. Then you should be able to follow the docs to get your Salt Cloud configuration files set up.

There's a ton of useful information about Salt Cloud in the Salt Cloud Docs.