Resolve merge conflicts

doc/ref/file_server/backends.rst

@@ -1,21 +1,43 @@
+.. _file-server-backends:
+
 ====================
 File Server Backends
 ====================
 
-Salt version 0.12.0 introduced the ability for the Salt Master to integrate
-different file server backends. File server backends allows the Salt file
-server to act as a transparent bridge to external resources. The primary
-example of this is the git backend which allows for all of the Salt formulas
-and files to be maintained in a remote git repository.
+In Salt 0.12.0, the modular fileserver was introduced. This feature added the
+ability for the Salt Master to integrate different file server backends. File
+server backends allow the Salt file server to act as a transparent bridge to
+external resources. A good example of this is the :mod:`git
+<salt.fileserver.git>` backend, which allows Salt to serve files sourced from
+one or more git repositories, but there are several others as well. Click
+:ref:`here <all-salt.fileserver>` for a full list of Salt's fileserver
+backends.
+
+Enabling a Fileserver Backend
+-----------------------------
+
+Fileserver backends can be enabled with the :conf_master:`fileserver_backend`
+option.
+
+.. code-block:: yaml
+
+    fileserver_backend:
+      - git
 
-The fileserver backend system can accept multiple backends as well. This makes
-it possible to have the environments listed in the :conf_master:`file_roots`
-configuration available in addition to other backends, or the ability to mix
-multiple backends.
+See the :ref:`documentation <all-salt.fileserver>` for each backend to find the
+correct value to add to :conf_master:`fileserver_backend` in order to enable
+them.
 
-This feature is managed by the :conf_master:`fileserver_backend` option in the
-master config. The desired backend systems are listed in order of search
-priority:
+Using Multiple Backends
+-----------------------
+
+If :conf_master:`fileserver_backend` is not defined in the Master config file,
+Salt will use the :mod:`roots <salt.fileserver.roots>` backend, but the
+:conf_master:`fileserver_backend` option supports multiple backends. When more
+than one backend is in use, the files from the enabled backends are merged into a
+single virtual filesystem. When a file is requested, the backends will be
+searched in order for that file, and the first backend to match will be the one
+which returns the file.
 
 .. code-block:: yaml
 
@@ -24,16 +46,56 @@ priority:
       - git
 
 With this configuration, the environments and files defined in the
-:conf_master:`file_roots` parameter will be searched first, if the referenced
-environment and file is not found then the :mod:`git <salt.fileserver.gitfs>`
-backend will be searched.
+:conf_master:`file_roots` parameter will be searched first, and if the file is
+not found then the git repositories defined in :conf_master:`gitfs_remotes`
+will be searched.
 
 Environments
 ------------
 
-The concept of environments is followed in all backend systems. The
-environments in the classic :mod:`roots <salt.fileserver.roots>` backend are
-defined in the :conf_master:`file_roots` option. Environments map differently
-based on the backend, for instance the git backend translated branches and tags
-in git to environments. This makes it easy to define environments in git by
-just setting a tag or forking a branch.
+Just as the order of the values in :conf_master:`fileserver_backend` matters,
+so too does the order in which different sources are defined within a
+fileserver environment. For example, given the below :conf_master:`file_roots`
+configuration, if both ``/srv/salt/dev/foo.txt`` and ``/srv/salt/prod/foo.txt``
+exist on the Master, then ``salt://foo.txt`` would point to
+``/srv/salt/dev/foo.txt`` in the ``dev`` environment, but it would point to
+``/srv/salt/prod/foo.txt`` in the ``base`` environment.
+
+.. code-block:: yaml
+
+    file_roots:
+      base:
+        - /srv/salt/prod
+      qa:
+        - /srv/salt/qa
+        - /srv/salt/prod
+      dev:
+        - /srv/salt/dev
+        - /srv/salt/qa
+        - /srv/salt/prod
+
+Similarly, when using the :mod:`git <salt.fileserver.gitfs>` backend, if both
+repositories defined below have a ``hotfix23`` branch/tag, and both of them
+also contain the file ``bar.txt`` in the root of the repository at that
+branch/tag, then ``salt://bar.txt`` in the ``hotfix23`` environment would be
+served from the ``first`` repository.
+
+.. code-block:: yaml
+
+    gitfs_remotes:
+      - https://mydomain.tld/repos/first.git
+      - https://mydomain.tld/repos/second.git
+
+.. note::
+
+    Environments map differently based on the fileserver backend. For instance,
+    the mappings are explicitly defined in :mod:`roots <salt.fileserver.roots>`
+    backend, while in the VCS backends (:mod:`git <salt.fileserver.gitfs>`,
+    :mod:`hg <salt.fileserver.hgfs>`, :mod:`svn <salt.fileserver.svnfs>`) the
+    environments are created from branches/tags/bookmarks/etc. For the
+    :mod:`minion <salt.fileserver.minionfs>` backend, the files are all in a
+    single environment, which is specified by the :conf_master:`minionfs_env`
+    option.
+
+    See the documentation for each backend for a more detailed explanation of
+    how environments are mapped.

doc/ref/output/all/index.rst

@@ -13,7 +13,6 @@ Follow one of the below links for further information and examples
     :template: autosummary.rst.tmpl
 
     compact
-    grains
     highstate
     json_out
     key

doc/topics/releases/2015.2.0.rst

@@ -32,6 +32,8 @@ Misc Fixes/Additions
   updates!)
 - Joyent now requires a ``keyname`` to be specified in the provider
   configuration. This change was necessitated upstream by the 7.0+ API.
+- Add ``args`` argument to ``cmd.script_retcode`` to match ``cmd.script`` in
+  the :py:mod:`cmd module <salt.cmd.cmdmod>`. (:issue:`21122`)
 
 Deprecations
 ============

doc/topics/tutorials/gitfs.rst

@@ -122,6 +122,30 @@ For APT-based distros such as Ubuntu and Debian:
 
     # apt-get install python-dulwich
 
+.. important::
+
+    If switching to Dulwich from GitPython/pygit2, or switching from
+    GitPython/pygit2 to Dulwich, it is necessary to clear the gitfs cache to
+    avoid unpredictable behavior. This is probably a good idea whenever
+    switching to a new :conf_master:`gitfs_provider`, but it is less important
+    when switching between GitPython and pygit2.
+
+    Beginning in version 2015.2.0, the gitfs cache can be easily cleared using
+    the :mod:`fileserver.clear_cache <salt.runners.fileserver.clear_cache>`
+    runner.
+
+    .. code-block:: bash
+
+        salt-run fileserver.clear_cache backend=git
+
+    If the Master is running an earlier version, then the cache can be cleared
+    by removing the ``gitfs`` and ``file_lists/gitfs`` directories (both paths
+    relative to the master cache directory, usually
+    ``/var/cache/salt/master``).
+
+    .. code-block:: bash
+
+        rm -rf /var/cache/salt/master{,/file_lists}/gitfs
 
 Simple Configuration
 ====================
@@ -157,6 +181,14 @@ master:
    Information on how to authenticate to SSH remotes can be found :ref:`here
    <gitfs-authentication>`.
 
+   .. note::
+
+       Dulwich does not recognize ``ssh://`` URLs, ``git+ssh://`` must be used
+       instead. Salt version 2015.2.0 and later will automatically add the
+       ``git+`` to the beginning of these URLs before fetching, but earlier
+       Salt versions will fail to fetch unless the URL is specified using
+       ``git+ssh://``.
+
 3. Restart the master to load the new configuration.
 
 

doc/topics/tutorials/lxc.rst

@@ -208,7 +208,23 @@ will directly correspond to a parameter in an LXC configuration file (see ``man
 - **flags** - Corresponds to **lxc.network.flags**
 
 Interface-specific options (MAC address, IPv4/IPv6, etc.) must be passed on a
-container-by-container basis.
+container-by-container basis, for instance using the ``nic_opts`` argument to
+:mod:`lxc.create <salt.modules.lxc.create>`:
+
+.. code-block:: bash
+
+    salt myminion lxc.create container1 profile=centos network_profile=centos nic_opts='{eth0: {ipv4: 10.0.0.20/24, gateway: 10.0.0.1}}'
+
+.. warning::
+
+    The ``ipv4``, ``ipv6``, ``gateway``, and ``link`` (bridge) settings in
+    network profiles / nic_opts will only work if the container doesnt redefine
+    the network configuration (for example in
+    ``/etc/sysconfig/network-scripts/ifcfg-<interface_name>`` on RHEL/CentOS,
+    or ``/etc/network/interfaces`` on Debian/Ubuntu/etc.). Use these with
+    caution. The container images installed using the ``download`` template,
+    for instance, typically are configured for eth0 to use DHCP, which will
+    conflict with static IP addresses set at the container level.
 
 
 Creating a Container on the CLI
@@ -404,15 +420,15 @@ New functions have been added to mimic the behavior of the functions in the
 equivalents:
 
 
-======================================= ====================================================== ===========================================================
+======================================= ====================================================== ===================================================
 Description                             :mod:`cmd <salt.modules.cmdmod>` module                :mod:`lxc <salt.modules.lxc>` module
-======================================= ====================================================== ===========================================================
-Run a command and get all output        :mod:`cmd.run <salt.modules.cmdmod.run>`               :mod:`lxc.cmd_run <salt.modules.lxc.cmd_run>`
-Run a command and get just stdout       :mod:`cmd.run_stdout <salt.modules.cmdmod.run_stdout>` :mod:`lxc.cmd_run_stdout <salt.modules.lxc.cmd_run_stdout>`
-Run a command and get just stderr       :mod:`cmd.run_stderr <salt.modules.cmdmod.run_stderr>` :mod:`lxc.cmd_run_stderr <salt.modules.lxc.cmd_run_stderr>`
-Run a command and get just the retcode  :mod:`cmd.retcode <salt.modules.cmdmod.retcode>`       :mod:`lxc.cmd_retcode <salt.modules.lxc.cmd_retcode>`
-Run a command and get all information   :mod:`cmd.run_all <salt.modules.cmdmod.run_all>`       :mod:`lxc.cmd_run_all <salt.modules.lxc.cmd_run_all>`
-======================================= ====================================================== ===========================================================
+======================================= ====================================================== ===================================================
+Run a command and get all output        :mod:`cmd.run <salt.modules.cmdmod.run>`               :mod:`lxc.run <salt.modules.lxc.run>`
+Run a command and get just stdout       :mod:`cmd.run_stdout <salt.modules.cmdmod.run_stdout>` :mod:`lxc.run_stdout <salt.modules.lxc.run_stdout>`
+Run a command and get just stderr       :mod:`cmd.run_stderr <salt.modules.cmdmod.run_stderr>` :mod:`lxc.run_stderr <salt.modules.lxc.run_stderr>`
+Run a command and get just the retcode  :mod:`cmd.retcode <salt.modules.cmdmod.retcode>`       :mod:`lxc.retcode <salt.modules.lxc.retcode>`
+Run a command and get all information   :mod:`cmd.run_all <salt.modules.cmdmod.run_all>`       :mod:`lxc.run_all <salt.modules.lxc.run_all>`
+======================================= ====================================================== ===================================================
 
 
 2014.7.x and Earlier

doc/topics/tutorials/states_pt4.rst

@@ -33,6 +33,18 @@ same relative path in more than one root, then the top-most match "wins". For
 example, if ``/srv/salt/foo.txt`` and ``/mnt/salt-nfs/base/foo.txt`` both
 exist, then ``salt://foo.txt`` will point to ``/srv/salt/foo.txt``.
 
+.. note::
+
+    When using multiple fileserver backends, the order in which they are listed
+    in the :conf_master:`fileserver_backend` parameter also matters. If both
+    ``roots`` and ``git`` backends contain a file with the same relative path,
+    and ``roots`` appears before ``git`` in the
+    :conf_master:`fileserver_backend` list, then the file in ``roots`` will
+    "win", and the file in gitfs will be ignored.
+
+    A more thorough explanation of how Salt's modular fileserver works can be
+    found :ref:`here <file-server-backends>`. We recommend reading this.
+
 Environment configuration
 =========================
 
@@ -192,4 +204,4 @@ who are using Salt, we have a very :ref:`active community <salt-community>`
 and we'd love to hear from you.
 
 In addition, by continuing to :doc:`part 5 <states_pt5>`, you can learn about
-the powerful orchestration of which Salt is capable.
+the powerful orchestration of which Salt is capable.

salt/client/ssh/__init__.py

@@ -490,7 +490,7 @@ def run(self):
         # Save the invocation information
         argv = self.opts['argv']
 
-        if self.opts['raw_shell']:
+        if self.opts.get('raw_shell', False):
             fun = 'ssh._raw'
             args = argv
         else:
@@ -687,7 +687,7 @@ def run(self, deploy_attempted=False):
         '''
         stdout = stderr = retcode = None
 
-        if self.opts.get('raw_shell'):
+        if self.opts.get('raw_shell', False):
             cmd_str = ' '.join([self._escape_arg(arg) for arg in self.argv])
             stdout, stderr, retcode = self.shell.exec_cmd(cmd_str)
 

salt/cloud/clouds/ec2.py

@@ -96,6 +96,7 @@
     import Crypto
     # PKCS1_v1_5 was added in PyCrypto 2.5
     from Crypto.Cipher import PKCS1_v1_5  # pylint: disable=E0611
+    from Crypto.Hash import SHA  # pylint: disable=E0611,W0611
     HAS_PYCRYPTO = True
 except ImportError:
     HAS_PYCRYPTO = False
@@ -2070,6 +2071,8 @@ def create(vm_=None, call=None):
             )
         )
     vm_['key_filename'] = key_filename
+    # wait_for_instance requires private_key
+    vm_['private_key'] = key_filename
 
     # Get SSH Gateway config early to verify the private_key,
     # if used, exists or not. We don't want to deploy an instance

salt/cloud/clouds/pyrax.py

@@ -17,7 +17,10 @@
 import salt.config as config
 
 # Import pyrax libraries
-import salt.utils.openstack.pyrax as suop
+# This is typically against SaltStack coding styles,
+# it should be 'import salt.utils.openstack.pyrax as suop'.  Something
+# in the loader is creating a name clash and making that form fail
+from salt.utils.openstack import pyrax as suop
 
 
 # Only load in this module is the OPENSTACK configurations are in place

salt/cloud/deploy/bootstrap-salt.sh

@@ -17,7 +17,7 @@
 #       CREATED: 10/15/2012 09:49:37 PM WEST
 #======================================================================================================================
 set -o nounset                              # Treat unset variables as an error
-__ScriptVersion="2015.02.27"
+__ScriptVersion="2015.02.28"
 __ScriptName="bootstrap-salt.sh"
 
 #======================================================================================================================
@@ -2039,13 +2039,9 @@ _eof
     fi
 
     # Debian Backports
-    if [ "$(grep -R 'backports.debian.org' /etc/apt)" = "" ]; then
-        echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> \
+    if [ "$(grep -R 'squeeze-backports' /etc/apt | grep -v "^#")" = "" ]; then
+        echo "deb http://http.debian.net/debian-backports squeeze-backports main" >> \
             /etc/apt/sources.list.d/backports.list
-
-        # Add the backports key
-        gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
-        gpg -a --export 8B48AD6246925553 | apt-key add -
     fi
 
     # Saltstack's Stable Debian repository
@@ -2098,6 +2094,12 @@ install_debian_7_deps() {
     # Install Keys
     __apt_get_install_noinput debian-archive-keyring && apt-get update
 
+    # Debian Backports
+    if [ "$(grep -R 'wheezy-backports' /etc/apt | grep -v "^#")" = "" ]; then
+        echo "deb http://http.debian.net/debian wheezy-backports main" >> \
+            /etc/apt/sources.list.d/backports.list
+    fi
+
     # Saltstack's Stable Debian repository
     if [ "$(grep -R 'wheezy-saltstack' /etc/apt)" = "" ]; then
         echo "deb http://debian.saltstack.com/debian wheezy-saltstack main" >> \

salt/daemons/masterapi.py

@@ -233,8 +233,10 @@ def fileserver_update(fileserver):
     '''
     try:
         if not fileserver.servers:
-            log.error('No fileservers loaded, the master will not be'
-                      'able to serve files to minions')
+            log.error(
+                'No fileservers loaded, the master will not be able to '
+                'serve files to minions'
+            )
             raise SaltMasterError('No fileserver backends available')
         fileserver.update()
     except Exception as exc:

salt/exceptions.py

@@ -80,6 +80,12 @@ class MinionError(SaltException):
     '''
 
 
+class FileserverConfigError(SaltException):
+    '''
+    Used when invalid fileserver settings are detected
+    '''
+
+
 class SaltInvocationError(SaltException, TypeError):
     '''
     Used when the wrong number of arguments are sent to modules or invalid