Merge branch '2016.3' into 2016_3_develop

doc/ref/configuration/master.rst

@@ -2315,9 +2315,9 @@ exposed.
 .. code-block:: yaml
 
     minionfs_whitelist:
-      - base
-      - v1.*
-      - 'mybranch\d+'
+      - server01
+      - dev*
+      - 'mail\d+.mydomain.tld'
 
 .. conf_master:: minionfs_blacklist
 
@@ -2341,9 +2341,9 @@ exposed.
 .. code-block:: yaml
 
     minionfs_blacklist:
-      - base
-      - v1.*
-      - 'mybranch\d+'
+      - server01
+      - dev*
+      - 'mail\d+.mydomain.tld'
 
 
 .. _pillar-configuration:

doc/topics/tutorials/minionfs.rst

@@ -4,155 +4,149 @@
 MinionFS Backend Walkthrough
 ============================
 
-Propagating Files
-=================
-
 .. versionadded:: 2014.1.0
 
-Sometimes, one might need to propagate files that are generated on a minion.
-Salt already has a feature to send files from a minion to the master.
+.. note::
 
-Enabling File Propagation
-=========================
+    This walkthrough assumes basic knowledge of Salt and :mod:`cp.push
+    <salt.modules.cp.push>`. To get up to speed, check out the
+    :doc:`walkthrough </topics/tutorials/walkthrough>`.
 
-To enable propagation, the :conf_master:`file_recv` option needs to be set to ``True``.
+Sometimes it is desirable to deploy a file located on one minion to one or more
+other minions. This is supported in Salt, and can be accomplished in two parts:
 
-.. code-block:: yaml
+#. Minion support for pushing files to the master (using :py:func:`cp.push
+   <salt.modules.cp.push>`)
 
-    file_recv: True
+#. The :mod:`minionfs <salt.fileserver.minionfs>` fileserver backend
 
-These changes require a restart of the master, then new requests for the
-``salt://minion-id/`` protocol will send files that are pushed by ``cp.push``
-from ``minion-id`` to the master.
 
-.. code-block:: bash
+This walkthrough will show how to use both of these features.
 
-    salt 'minion-id' cp.push /path/to/the/file
 
-This command will store the file, including its full path, under
-:conf_master:`cachedir` ``/master/minions/minion-id/files``. With the default
-:conf_master:`cachedir` the example file above would be stored as
-`/var/cache/salt/master/minions/minion-id/files/path/to/the/file`.
+Enabling File Push
+==================
 
-.. note::
+To set the master to accept files pushed from minions, the
+:conf_master:`file_recv` option in the master config file must be set to
+``True`` (the default is ``False``).
 
-    This walkthrough assumes basic knowledge of Salt and :mod:`cp.push
-    <salt.modules.cp.push>`. To get up to speed, check out the
-    :doc:`walkthrough </topics/tutorials/walkthrough>`.
+.. code-block:: yaml
 
-MinionFS Backend
-================
+    file_recv: True
 
-Since it is not a good idea to expose the whole :conf_master:`cachedir`, MinionFS
-should be used to send these files to other minions.
+.. note::
+    This change requires a restart of the salt-master service.
 
-Simple Configuration
-====================
+Pushing Files
+=============
 
-To use the minionfs backend only two configuration changes are required on the
-master. The :conf_master:`fileserver_backend` option needs to contain a value of
-``minion`` and :conf_master:`file_recv` needs to be set to true:
+Once this has been done, files can be pushed to the master using the
+:py:func:`cp.push <salt.modules.cp.push>` function:
 
-.. code-block:: yaml
-
-    fileserver_backend:
-      - roots
-      - minion
+.. code-block:: bash
 
-    file_recv: True
+    salt 'minion-id' cp.push /path/to/the/file
 
-These changes require a restart of the master, then new requests for the
-``salt://minion-id/`` protocol will send files that are pushed by ``cp.push``
-from ``minion-id`` to the master.
+This command will store the file in a subdirectory named ``minions`` under the
+master's :conf_master:`cachedir`. On most masters, this path will be
+``/var/cache/salt/master/minions``. Within this directory will be one directory
+for each minion which has pushed a file to the master, and underneath that the
+full path to the file on the minion. So, for example, if a minion with an ID of
+``dev1`` pushed a file ``/var/log/myapp.log`` to the master, it would be saved
+to ``/var/cache/salt/master/minions/dev1/var/log/myapp.log``.
 
-.. note::
+Serving Pushed Files Using MinionFS
+===================================
 
-    All of the files that are pushed to the master are going to be available to
-    all of the minions. If this is not what you want, please remove ``minion``
-    from :conf_master:`fileserver_backend` in the master config file.
+While it is certainly possible to add ``/var/cache/salt/master/minions`` to the
+master's :conf_master:`file_roots` and serve these files, it may only be
+desirable to expose files pushed from certain minions. Adding
+``/var/cache/salt/master/minions/<minion-id>`` for each minion that needs to be
+exposed can be cumbersome and prone to errors.
 
-.. note::
+Enter :mod:`minionfs <salt.fileserver.minionfs>`. This fileserver backend will
+make files pushed using :py:func:`cp.push <salt.modules.cp.push>` available to
+the Salt fileserver, and provides an easy mechanism to restrict which minions'
+pushed files are made available.
 
-    Having directories with the same name as your minions in the root
-    that can be accessed like ``salt://minion-id/`` might cause confusion.
+Simple Configuration
+--------------------
 
-Commandline Example
-===================
+To use the :mod:`minionfs <salt.fileserver.minionfs>` backend, add ``minion``
+to the list of backends in the :conf_master:`fileserver_backend` configuration
+option on the master:
 
-Lets assume that we are going to generate SSH keys on a minion called
-``minion-source`` and put the public part in ``~/.ssh/authorized_keys`` of root
-user of a minion called ``minion-destination``.
+.. code-block:: yaml
 
-First, lets make sure that ``/root/.ssh`` exists and has the right permissions:
+    file_recv: True
 
-.. code-block:: bash
+    fileserver_backend:
+      - roots
+      - minion
 
-    [root@salt-master file]# salt '*' file.mkdir dir_path=/root/.ssh user=root group=root mode=700
-    minion-source:
-        None
-    minion-destination:
-        None
+.. note::
+    As described earlier, ``file_recv: True`` is also needed to enable the
+    master to receive files pushed from minions. As always, changes to the
+    master configuration require a restart of the ``salt-master`` service.
 
-We create an RSA key pair without a passphrase [*]_:
+Files made available via :mod:`minionfs <salt.fileserver.minionfs>` are by
+default located at ``salt://<minion-id>/path/to/file``. Think back to the
+earlier example, in which ``dev1`` pushed a file ``/var/log/myapp.log`` to the
+master. With :mod:`minionfs <salt.fileserver.minionfs>` enabled, this file
+would be addressable in Salt at ``salt://dev1/var/log/myapp.log``.
 
-.. code-block:: bash
+If many minions have pushed to the master, this will result in many directories
+in the root of the Salt fileserver. For this reason, it is recommended to use
+the :conf_master:`minionfs_mountpoint` config option to organize these files
+underneath a subdirectory:
 
-    [root@salt-master file]# salt 'minion-source' cmd.run 'ssh-keygen -N "" -f /root/.ssh/id_rsa'
-    minion-source:
-        Generating public/private rsa key pair.
-        Your identification has been saved in /root/.ssh/id_rsa.
-        Your public key has been saved in /root/.ssh/id_rsa.pub.
-        The key fingerprint is:
-        9b:cd:1c:b9:c2:93:8e:ad:a3:52:a0:8b:0a:cc:d4:9b root@minion-source
-        The key's randomart image is:
-        +--[ RSA 2048]----+
-        |                 |
-        |                 |
-        |                 |
-        |  o        .     |
-        | o o    S o      |
-        |=   +  . B o     |
-        |o+ E    B =      |
-        |+ .   .+ o       |
-        |o  ...ooo        |
-        +-----------------+
-
-and we send the public part to the master to be available to all minions:
+.. code-block:: yaml
 
-.. code-block:: bash
+    minionfs_mountpoint: salt://minionfs
 
-    [root@salt-master file]# salt 'minion-source' cp.push /root/.ssh/id_rsa.pub
-    minion-source:
-        True
+Using the above mountpoint, the file in the example would be located at
+``salt://minionfs/dev1/var/log/myapp.log``.
 
-now it can be seen by everyone:
 
-.. code-block:: bash
+Restricting Certain Minions' Files from Being Available Via MinionFS
+--------------------------------------------------------------------
 
-    [root@salt-master file]# salt 'minion-destination' cp.list_master_dirs
-    minion-destination:
-        - .
-        - etc
-        - minion-source/root
-        - minion-source/root/.ssh
+A whitelist and blacklist can be used to restrict the minions whose pushed
+files are available via :mod:`minionfs <salt.fileserver.minionfs>`. These lists
+can be managed using the :conf_master:`minionfs_whitelist` and
+:conf_master:`minionfs_blacklist` config options. Click the links for both of
+them for a detailed explanation of how to use them.
 
-Lets copy that as the only authorized key to ``minion-destination``:
+A more complex configuration example, which uses both a whitelist and
+blacklist, can be found below:
 
-.. code-block:: bash
+.. code-block:: yaml
 
-    [root@salt-master file]# salt 'minion-destination' cp.get_file salt://minion-source/root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
-    minion-destination:
-        /root/.ssh/authorized_keys
+    file_recv: True
 
-Or we can use a more elegant and salty way to add an SSH key:
+    fileserver_backend:
+      - roots
+      - minion
 
-.. code-block:: bash
+    minionfs_mountpoint: salt://minionfs
 
-    [root@salt-master file]# salt 'minion-destination' ssh.set_auth_key_from_file user=root source=salt://minion-source/root/.ssh/id_rsa.pub
-    minion-destination:
-        new
+    minionfs_whitelist:
+      - host04
+      - web*
+      - 'mail\d+\.domain\.tld'
 
+    minionfs_whitelist:
+      - web21
 
+Potential Concerns
+------------------
 
+* There is no access control in place to restrict which minions have access to
+  files served up by :mod:`minionfs <salt.fileserver.minionfs>`. All minions
+  will have access to these files.
 
-.. [*] Yes, that was the actual key on my server, but the server is already destroyed.
+* Unless the :conf_master:`minionfs_whitelist` and/or
+  :conf_master:`minionfs_blacklist` config options are used, all minions which
+  push files to the master will have their files made available via
+  :mod:`minionfs <salt.fileserver.minionfs>`.

salt/log/setup.py

@@ -798,7 +798,6 @@ def setup_multiprocessing_logging_listener(opts, queue=None):
         target=__process_multiprocessing_logging_queue,
         args=(opts, queue or get_multiprocessing_logging_queue(),)
     )
-    __MP_LOGGING_QUEUE_PROCESS.daemon = True
     __MP_LOGGING_QUEUE_PROCESS.start()
     __MP_LOGGING_LISTENER_CONFIGURED = True
 

salt/master.py

@@ -539,7 +539,8 @@ def start(self):
                                       'reload': salt.crypt.Crypticle.generate_key_string
                                      }
             log.info('Creating master process manager')
-            self.process_manager = salt.utils.process.ProcessManager(name='Master_ProcessManager')
+            # Since there are children having their own ProcessManager we should wait for kill more time.
+            self.process_manager = salt.utils.process.ProcessManager(wait_for_kill=5)
             pub_channels = []
             log.info('Creating master publisher process')
             for transport, opts in iter_transport_opts(self.opts):
@@ -726,7 +727,9 @@ def __bind(self):
             except os.error:
                 pass
 
-        self.process_manager = salt.utils.process.ProcessManager(name='ReqServer_ProcessManager')
+        # Wait for kill should be less then parent's ProcessManager.
+        self.process_manager = salt.utils.process.ProcessManager(name='ReqServer_ProcessManager',
+                                                                 wait_for_kill=1)
 
         req_channels = []
         tcp_only = True

salt/modules/archive.py

@@ -6,6 +6,7 @@
 '''
 from __future__ import absolute_import
 import os
+import logging
 import contextlib  # For < 2.7 compat
 import logging
 
@@ -21,6 +22,8 @@
     'zip_': 'zip'
 }
 
+log = logging.getLogger(__name__)
+
 
 HAS_ZIPFILE = False
 try:
@@ -535,8 +538,12 @@ def unzip(zip_file, dest, excludes=None, options=None, template=None,
 
         salt '*' archive.unzip /tmp/zipfile.zip /home/strongbad/ password='BadPassword'
     '''
-    if options:
-        log.warning("Options '{0}' ignored, only works with unzip binary.".format(options))
+    # https://bugs.python.org/issue15795
+    log.warning('Due to bug 15795 in python\'s zip lib, the permissions of the'
+                ' extracted files may not be preserved when using archive.unzip')
+    log.warning('To preserve the permissions of extracted files, use'
+                ' archive.cmd_unzip')
+
     if not excludes:
         excludes = []
     if runas: