Merge pull request #25271 from basepi/merge-forward-2015.8

[2015.8] Merge forward from 2015.5 to 2015.8

doc/topics/cloud/gce.rst

@@ -49,13 +49,20 @@ Google Compute Engine Setup
    To set up authorization, navigate to *APIs & auth* section and then the
    *Credentials* link and click the *CREATE NEW CLIENT ID* button. Select
    *Service Account* and click the *Create Client ID* button. This will
-   automatically download a ``.json`` file, which should be ignored. Look for
-   a new *Service Account* section in the page and record the generated email
+   automatically download a ``.json`` file which can be ignored.
+
+   Look for a new *Service Account* section in the page and record the generated email
    address for the matching key/fingerprint. The email address will be used
    in the ``service_account_email_address`` of the ``/etc/salt/cloud`` file.
 
 #. Key Format
 
+   *If you are using ``libcloud >= 0.17.0`` it is recommended that you use the ``JSON 
+   format`` file you downloaded above and skip to the "Configuration" section below, using 
+   the JSON file **_in place of 'NEW.pem'_** in the documentation.
+   If you are using an older version of libcloud or are unsure of the version you 
+   have, please follow the instructions below to generate and format a new P12 key.*
+
    In the new *Service Account* section, click *Generate new P12 key*, which
    will automatically download a ``.p12`` private key file. The ``.p12``
    private key needs to be converted to a format compatible with libcloud.
@@ -69,7 +76,7 @@ Google Compute Engine Setup
        openssl pkcs12 -in ORIG.p12 -passin pass:notasecret \
        -nodes -nocerts | openssl rsa -out NEW.pem
 
-
+ 
 
 Configuration
 =============

doc/topics/eauth/index.rst

@@ -4,7 +4,7 @@
 External Authentication System
 ==============================
 
-Salt's External Authentication System (eAuth) allows for Salt to  pass through
+Salt's External Authentication System (eAuth) allows for Salt to pass through
 command authorization to any external authentication system, such as PAM or LDAP.
 
 .. note::
@@ -179,8 +179,8 @@ the master config:
 
 To determine group membership in AD, the username and password that is entered
 when LDAP is requested as the eAuth mechanism on the command line is used to
-bind to AD's LDAP interface.  If this fails, then it doesn't matter what groups
-the user belongs to, he or she is denied access.  Next, the distinguishedName
+bind to AD's LDAP interface. If this fails, then it doesn't matter what groups
+the user belongs to, he or she is denied access. Next, the distinguishedName
 of the user is looked up with the following LDAP search:
 
 .. code-block:: text
@@ -190,7 +190,7 @@ of the user is looked up with the following LDAP search:
     )
 
 This should return a distinguishedName that we can use to filter for group
-membership.  Then the following LDAP quey is executed:
+membership. Then the following LDAP query is executed:
 
 .. code-block:: text
 

doc/topics/releases/releasecandidate.rst

@@ -4,8 +4,8 @@ Installing/Testing a Salt Release Candidate
 
 It's time for a new feature release of Salt!  Follow the instructions below to
 install the latest release candidate of Salt, and try :doc:`all the shiny new
-features </topics/releases/2015.2.0>`!  Be sure to report any bugs you find on
-`GitHub <http://www.github.com/saltstack/salt>`_
+features </topics/releases/2015.8.0>`!  Be sure to report any bugs you find on
+`Github <http://www.github.com/saltstack/salt>`_
 
 Installing Using Bootstrap
 ==========================
@@ -16,23 +16,23 @@ The easiest way to install a release candidate of Salt is using
 .. code-block:: bash
 
     curl -o install_salt.sh -L https://bootstrap.saltstack.com
-    sudo sh install_salt.sh git v2015.2.0rc2
+    sudo sh install_salt.sh git v2015.8.0rc1
 
 If you want to also install a master using `Salt Bootstrap`_, use the ``-M``
 flag:
 
 .. code-block:: bash
 
     curl -o install_salt.sh -L https://bootstrap.saltstack.com
-    sudo sh install_salt.sh -M git v2015.2.0rc2
+    sudo sh install_salt.sh -M git v2015.8.0rc1
 
 If you want to install only a master and not a minion using `Salt Bootstrap`_,
 use the ``-M`` and ``-N`` flags:
 
 .. code-block:: bash
 
     curl -o install_salt.sh -L https://bootstrap.saltstack.com
-    sudo sh install_salt.sh -M -N git v2015.2.0rc2
+    sudo sh install_salt.sh -M -N git v2015.8.0rc1
 
 
 Installation from Source Tarball
@@ -44,9 +44,9 @@ installation docs <_installation>`.  Then install salt using the following:
 
 .. code-block:: bash
 
-    curl -O https://pypi.python.org/packages/source/s/salt/salt-2015.2.0rc2.tar.gz
-    tar -xzvf salt-2015.2.0rc2.tar.gz
-    cd salt-2015.2.0rc2
+    curl -O https://pypi.python.org/packages/source/s/salt/salt-2015.8.0rc1.tar.gz
+    tar -xzvf salt-2015.8.0rc1.tar.gz
+    cd salt-2015.8.0rc1
     sudo python setup.py install
 
 

doc/topics/tutorials/http.rst

@@ -68,8 +68,8 @@ be overridden with the ``method`` argument:
 
     salt.utils.http.query('http://example.com/delete/url', 'DELETE')
 
-When using the ``POST`` method (and others, such ``PUT``), extra data is usually
-sent as well. This data can be either sent directly, in whatever format is
+When using the ``POST`` method (and others, such as ``PUT``), extra data is usually
+sent as well. This data can be sent directly, in whatever format is
 required by the remote server (XML, JSON, plain text, etc).
 
 .. code-block:: python
@@ -146,8 +146,8 @@ However, this can be changed to ``master`` if necessary.
     )
 
 Headers may also be passed through, either as a ``header_list``, a
-``header_dict`` or as a ``header_file``. As with the ``data_file``, the
-``header_file`` may also  be templated. Take note that because HTTP headers are
+``header_dict``, or as a ``header_file``. As with the ``data_file``, the
+``header_file`` may also be templated. Take note that because HTTP headers are
 normally syntactically-correct YAML, they will automatically be imported as an
 a Python dict.
 

salt/payload.py

@@ -120,7 +120,7 @@ def dumps(self, msg):
         '''
         try:
             return msgpack.dumps(msg)
-        except OverflowError:
+        except (OverflowError, msgpack.exceptions.PackValueError):
             # msgpack can't handle the very long Python longs for jids
             # Convert any very long longs to strings
             # We borrow the technique used by TypeError below

salt/states/archive.py

@@ -206,18 +206,17 @@ def extracted(name,
 
     __salt__['file.makedirs'](name, user=user, group=group)
 
-    if archive_format in ('zip', 'rar'):
-        log.debug('Extract {0} in {1}'.format(filename, name))
-        files = __salt__['archive.un{0}'.format(archive_format)](filename,
-                                                                 name)
+    log.debug('Extract {0} in {1}'.format(filename, name))
+    if archive_format == 'zip':
+        files = __salt__['archive.cmd_unzip'](filename, name)
+    elif archive_format == 'rar':
+        files = __salt__['archive.unrar'](filename, name)
     else:
         if tar_options is None:
             with closing(tarfile.open(filename, 'r')) as tar:
                 files = tar.getnames()
                 tar.extractall(name)
         else:
-            log.debug('Untar {0} in {1}'.format(filename, name))
-
             tar_opts = tar_options.split(' ')
 
             tar_cmd = ['tar']

tests/unit/modules/boto_vpc_test.py

@@ -24,7 +24,7 @@
 
 # Import 3rd-party libs
 import salt.ext.six as six
-# pylint: disable=import-error
+# pylint: disable=import-error,no-name-in-module
 try:
     import boto
     from boto.exception import BotoServerError
@@ -33,6 +33,7 @@
     HAS_BOTO = False
 
 try:
+    import moto
     from moto import mock_ec2
     HAS_MOTO = True
 except ImportError:
@@ -50,7 +51,7 @@ def stub_function(self):
             pass
 
         return stub_function
-# pylint: enable=import-error
+# pylint: enable=import-error,no-name-in-module
 
 # the boto_vpc module relies on the connect_to_region() method
 # which was added in boto 2.8.0
@@ -96,13 +97,17 @@ def _has_required_moto():
     if not HAS_MOTO:
         return False
     else:
-        import pkg_resources
-        from pkg_resources import DistributionNotFound
         try:
-            if LooseVersion(pkg_resources.get_distribution('moto').version) < LooseVersion(required_moto_version):
+            if LooseVersion(moto.__version__) < LooseVersion(required_moto_version):
+                return False
+        except AttributeError:
+            import pkg_resources
+            from pkg_resources import DistributionNotFound
+            try:
+                if LooseVersion(pkg_resources.get_distribution('moto').version) < LooseVersion(required_moto_version):
+                    return False
+            except DistributionNotFound:
                 return False
-        except DistributionNotFound:
-            return False
         return True
 
 
@@ -224,6 +229,7 @@ def _create_route_table(self, vpc_id, name=None, tags=None):
 @skipIf(_has_required_boto() is False, 'The boto module must be greater than'
                                        ' or equal to version {0}'
         .format(required_boto_version))
+@skipIf(_has_required_moto() is False, 'The moto version must be >= to version {0}'.format(required_moto_version))
 class BotoVpcTestCase(BotoVpcTestCaseBase, BotoVpcTestCaseMixin):
     '''
     TestCase for salt.modules.boto_vpc module
@@ -241,8 +247,6 @@ def test_that_when_checking_if_a_vpc_exists_by_id_and_a_vpc_exists_the_vpc_exist
         self.assertTrue(vpc_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering vpcs.'
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_vpc_exists_by_id_and_a_vpc_does_not_exist_the_vpc_exists_method_returns_false(
             self):
         '''
@@ -266,8 +270,6 @@ def test_that_when_checking_if_a_vpc_exists_by_name_and_a_vpc_exists_the_vpc_exi
         self.assertTrue(vpc_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering vpcs.'
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_vpc_exists_by_name_and_a_vpc_does_not_exist_the_vpc_exists_method_returns_false(
             self):
         '''
@@ -291,8 +293,6 @@ def test_that_when_checking_if_a_vpc_exists_by_tags_and_a_vpc_exists_the_vpc_exi
         self.assertTrue(vpc_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering vpcs.'
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_vpc_exists_by_tags_and_a_vpc_does_not_exist_the_vpc_exists_method_returns_false(
             self):
         '''
@@ -316,8 +316,6 @@ def test_that_when_checking_if_a_vpc_exists_by_cidr_and_a_vpc_exists_the_vpc_exi
         self.assertTrue(vpc_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering vpcs.'
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_vpc_exists_by_cidr_and_a_vpc_does_not_exist_the_vpc_exists_method_returns_false(
             self):
         '''
@@ -539,6 +537,7 @@ def test_that_when_describing_vpc_but_providing_no_vpc_id_the_describe_method_ra
 @skipIf(_has_required_boto() is False, 'The boto module must be greater than'
                                        ' or equal to version {0}'
         .format(required_boto_version))
+@skipIf(_has_required_moto() is False, 'The moto version must be >= to version {0}'.format(required_moto_version))
 class BotoVpcSubnetsTestCase(BotoVpcTestCaseBase, BotoVpcTestCaseMixin):
     @mock_ec2
     def test_get_subnet_association_single_subnet(self):
@@ -668,8 +667,6 @@ def test_that_when_a_subnet_does_not_exist_the_subnet_exists_method_returns_fals
         self.assertFalse(subnet_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering by tags. '
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_subnet_exists_by_name_the_subnet_exists_method_returns_true(self):
         '''
         Tests checking subnet existence by name
@@ -682,8 +679,6 @@ def test_that_when_checking_if_a_subnet_exists_by_name_the_subnet_exists_method_
         self.assertTrue(subnet_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering by tags. '
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_subnet_exists_by_name_the_subnet_does_not_exist_the_subnet_method_returns_false(self):
         '''
         Tests checking subnet existence by name when it doesn't exist
@@ -696,8 +691,6 @@ def test_that_when_checking_if_a_subnet_exists_by_name_the_subnet_does_not_exist
         self.assertFalse(subnet_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering by tags. '
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_subnet_exists_by_tags_the_subnet_exists_method_returns_true(self):
         '''
         Tests checking subnet existence by tag
@@ -710,8 +703,6 @@ def test_that_when_checking_if_a_subnet_exists_by_tags_the_subnet_exists_method_
         self.assertTrue(subnet_exists_result['exists'])
 
     @mock_ec2
-    @skipIf(_has_required_moto() is False, 'The moto module does not support filtering by tags. '
-                                           'Added support in spulec/moto#218. Next release should solve this issue.')
     def test_that_when_checking_if_a_subnet_exists_by_tags_the_subnet_does_not_exist_the_subnet_method_returns_false(self):
         '''
         Tests checking subnet existence by tag when subnet doesn't exist
@@ -899,8 +890,7 @@ def test_that_when_a_subnet_does_not_exist_the_subnet_exists_method_returns_fals
 @skipIf(_has_required_boto() is False, 'The boto module must be greater than'
                                        ' or equal to version {0}'
         .format(required_boto_version))
-@skipIf(_has_required_moto() is False, 'The moto module has a bug in creating DHCP options which is fixed '
-                                       'in spulec/moto#214. Next release should solve this issue.')
+@skipIf(_has_required_moto() is False, 'The moto version must be >= to version {0}'.format(required_moto_version))
 class BotoVpcDHCPOptionsTestCase(BotoVpcTestCaseBase, BotoVpcTestCaseMixin):
     @mock_ec2
     def test_that_when_creating_dhcp_options_succeeds_the_create_dhcp_options_method_returns_true(self):