Catch provider errors in salt cloud

[cachedout]

mp@silver /etc/salt % sudo salt-cloud -p linode_1024 d1  
[INFO    ] salt-cloud starting
[INFO    ] Creating Cloud VM d1
[ERROR   ] Error creating d1 on LINODE

The following exception was thrown by libcloud when trying to run the initial deployment: 
A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.
Traceback (most recent call last):
  File "/home/mp/Devel/salt/salt/cloud/clouds/linode.py", line 194, in create
    data = conn.create_node(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/compute/drivers/linode.py", line 313, in create_node
    self.connection.request(API_ROOT, params=params)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 675, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/linode.py", line 105, in __init__
    raise self.errors[0]
LinodeException: (8) A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.
Error: There was a profile error: Failed to deploy VM

Catch and format in a friendlier way.

[UtahDave]

This happens all over inside salt-cloud

[cachedout]

K. Good to know. I'll clean it up. Thanks @UtahDave!

[gtmanfred]

👍

[luciddr34m3r]

I'd like to add that when you are running salt-cloud with a map file, tons of these exceptions cause the entire script to abort. If you are running in parallel mode, it bails out on installing salt on any machines that were being spun up, and you end up with nodes that have booted up, and no way of knowing which of them have salt installed (other than re-running your map, comparing the list of running nodes to the list of nodes that respond to a test.ping).

In fact, there are tons of reasons why salt-cloud will boot a minion, but the salt install will fail, and it would be really nice if there were a way to easily get that list. As it stands, I have to go manually run the list through something like excel, and manually delete and then re-run my map file. It's either that or manually install salt on those boxes. A more elegant solution would be VERY welcome.

[cachedout]

@luciddr34m3r Excellent feedback. Would you mind taking that second point regarding the inability to see failed bootstraps and turning it into a separate GitHub issue? I think it's different enough from the original problem that we should tackle them independently. Thanks!

[luciddr34m3r]

@cachedout Done. #14659

[cachedout]

@luciddr34m3r Thanks!

[earthgecko]

Just to point out we are having similar issues with libcloud and linode, not libcloud per se, more linode.

We have poked linode to try and get them to make their label and name for linodes to be RFC 1034/1035/2181 complaint in terms of what a hostname can be (1 to 63 chars). Programmatically bifurcating too match linodes notion of label/name in relation to what it is, a hostname has proved painful a number of times. We bump into it now and again. Perhaps if saltstack and libcloud communities made a shout out to linode, maybe linode would move on this.

[scub]

Good Evening,

At this time we are currently in the process of reviewing what changes are required in order to specifically support this functionality on our platform. While we cannot currently provide an answer at this time, we are aware of this issue, and are actively looking to rectify this.

While greater support from saltstack, and libcloud would strengthen the case for this mitigation, at this time it would be un-needed as we are already seeking to quash this in the coming weeks.

Penny
Linode Change Control Board

[notpeter]

Ran into this again today. There's also an icky side effect when it fails like this, there's a "Brand New" linode left in your account cause the rename is what failed, not the creation. Should Salt Cloud try and cleanup instances which haven't successfully built?

[techhat]

Deleting nodes, by default, that have just been created seems like a bad default behavior to me. But I would not be opposed to allowing an option, if you can think of a good way to do it.

[rallytime]

This particular stacktrace is now being caught for both the libcloud and linode-python versions of the Linode driver (tested with v2015.5.5).

Libcloud:

# salt-cloud -p linode-ubuntu-14 nt
[INFO    ] salt-cloud starting
[INFO    ] Creating Cloud VM nt
[ERROR   ] Error creating nt on Linode via Apache Libcloud

The following exception was thrown by libcloud when trying to run the initial deployment:
(8) A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.
Error: There was a profile error: Failed to deploy VM

Linode-Python:

# salt-cloud -p linode-ubuntu-14 nt
[INFO    ] salt-cloud starting
[INFO    ] Creating Cloud VM nt
[ERROR   ] There was a profile error: [{u'ERRORCODE': 8, u'ERRORMESSAGE': u'A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.'}]

However, when this driver was rewritten in 2015.8 to use the Linode API directly, the API doesn't appear to throw an exception if the label doesn't match Linode's label specs. The create function continues until completion. The side-effect of this is similar to the one described above: A Linode is created with a linode id such as linode1234567, which is bootstrapped with and minion and "Running", but the vm name is never assigned to the linode label, so you cannot target it with salt. I'm looking into fixing this.

[rallytime]

I have fixed this bug by implementing a check to _validate_name in the create function before the provisioning event is fired. This will log an error and exit if the provided VM name doesn't meet Linode's label standards and also avoids the nasty caveat where random linode1234567 instances are created in their various states on both the 2015.5 branch (stays in "Brand New" state) and the 2015.8 branch and newer (gets to the "Running" state and the minion is bootstrapped, but the master can't reach it because the label never gets set).

Here's a sample output of an invalid name:

# salt-cloud -p linode-ubuntu-14 nt
[INFO    ] salt-cloud starting
[WARNING ] A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.
Error: There was a profile error: Failed to deploy VM

Where no extraneous linodes were created.

See #26709 for more information.