Add auto_detect feature to ssh_ext_alternatives

doc/topics/releases/sodium.rst

@@ -44,11 +44,11 @@ Salt-SSH updates
 ssh_pre_flight
 --------------
 
-A new Salt-SSH roster option `ssh_pre_flight` has been added. This enables you to run a
+A new Salt-SSH roster option ``ssh_pre_flight`` has been added. This enables you to run a
 script before Salt-SSH tries to run any commands. You can set this option in the roster
-for a specific minion or use the `roster_defaults` to set it for all minions.
+for a specific minion or use the ``roster_defaults`` to set it for all minions.
 
-Example for setting `ssh_pre_flight` for specific host in roster file
+Example for setting ``ssh_pre_flight`` for specific host in roster file
 
 .. code-block:: yaml
 
@@ -58,15 +58,15 @@ Example for setting `ssh_pre_flight` for specific host in roster file
     passwd: P@ssword
     ssh_pre_flight: /srv/salt/pre_flight.sh
 
-Example for setting `ssh_pre_flight` using roster_defaults, so all minions
+Example for setting ``ssh_pre_flight`` using roster_defaults, so all minions
 run this script.
 
 .. code-block:: yaml
 
   roster_defaults:
     ssh_pre_flight: /srv/salt/pre_flight.sh
 
-The `ssh_pre_flight` script will only run if the thin dir is not currently on the
+The ``ssh_pre_flight`` script will only run if the thin dir is not currently on the
 minion. If you want to force the script to run you have the following options:
 
 * Wipe the thin dir on the targeted minion using the -w arg.
@@ -89,6 +89,31 @@ You can set this setting in your roster file like so:
     set_path: '$PATH:/usr/local/bin/'
 
 
+auto_detect
+-----------
+
+You can now auto detect the dependencies to be packed into the salt thin when using
+the ``ssh_ext_alternatives`` feature.
+
+.. code-block:: yaml
+
+       ssh_ext_alternatives:
+           2019.2:                     # Namespace, can be anything.
+               py-version: [2, 7]      # Constraint to specific interpreter version
+               path: /opt/2019.2/salt  # Main Salt installation directory.
+               auto_detect: True       # Auto detect dependencies
+               py_bin: /usr/bin/python2.7 # Python binary path used to auto detect dependencies
+
+This new ``auto_detect`` option needs to be set to True in your ``ssh_ext_alternatives`` configuration.
+Salt-ssh will attempt to auto detect the file paths required for the default dependencies to include
+in the thin. If you have a dependency already set in your configuration, it will not attempt to auto
+detect for that dependency.
+
+You can also set the ``py_bin`` option to set the python binary to be used to auto detect the
+dependencies. If ``py_bin`` is not set, it will attempt to use the major Python version set in
+``py-version``. For example, if you set ``py-version`` to be ``[2, 7]`` it will attempt to find and
+use the ``python2`` binary.
+
 State Changes
 =============
 - Adding a new option for the State compiler, ``disabled_requisites`` will allow

doc/topics/ssh/index.rst

@@ -311,3 +311,13 @@ It is recommended that one modify this command a bit by removing the ``-l quiet`
 .. toctree::
 
     roster
+    ssh_ext_alternatives
+
+Different Python Versions
+=========================
+The Sodium release removed python 2 support in Salt. Even though this python 2 support
+is being dropped we have provided multiple ways to work around this with Salt-SSH. You
+can use the following options:
+
+  * :ref:`ssh_pre_flight <ssh_pre_flight>`
+  * :ref:`SSH ext alternatives <ssh-ext-alternatives>`

doc/topics/ssh/ssh_ext_alternatives.rst

@@ -0,0 +1,70 @@
+.. _ssh-ext-alternatives:
+
+====================
+SSH Ext Alternatives
+====================
+
+In the 2019.2.0 release the ``ssh_ext_alternatives`` feature was added.
+This allows salt-ssh to work across different python versions. You will
+need to ensure you have the following:
+
+  - Salt is installed, with all required dependnecies for both Python2 and Python3
+  - Everything needs to be importable from the respective Python environment.
+
+To enable using this feature you will need to edit the master configuration similar
+to below:
+
+.. code-block:: yaml
+
+       ssh_ext_alternatives:
+           2019.2:                     # Namespace, can be anything.
+               py-version: [2, 7]      # Constraint to specific interpreter version
+               path: /opt/2019.2/salt  # Main Salt installation directory.
+               dependencies:           # List of dependencies and their installation paths
+                 jinja2: /opt/jinja2
+                 yaml: /opt/yaml
+                 tornado: /opt/tornado
+                 msgpack: /opt/msgpack
+                 certifi: /opt/certifi
+                 singledispatch: /opt/singledispatch.py
+                 singledispatch_helpers: /opt/singledispatch_helpers.py
+                 markupsafe: /opt/markupsafe
+                 backports_abc: /opt/backports_abc.py
+
+auto_detect
+-----------
+
+In the Sodium release the ``auto_detect`` feature was added for ``ssh_ext_alternatives``.
+This allows salt-ssh to automatically detect the path to all of your dependencies and
+does not require you to define them under ``dependencies``.
+
+.. code-block:: yaml
+
+       ssh_ext_alternatives:
+           2019.2:                     # Namespace, can be anything.
+               py-version: [2, 7]      # Constraint to specific interpreter version
+               path: /opt/2019.2/salt  # Main Salt installation directory.
+               auto_detect: True       # Auto detect dependencies
+               py_bin: /usr/bin/python2.7 # Python binary path used to auto detect dependencies
+
+If ``py_bin`` is not set alongside ``auto_detect``, it will attempt to auto detect
+the dependnecies using the major version set in ``py-version``. For example if you
+have ``[2, 7]`` set as your ``py-version``, it will attempt to use the binary ``python2``.
+
+You can also use ``auto_detect`` and ``dependencies`` together.
+
+.. code-block:: yaml
+
+       ssh_ext_alternatives:
+           2019.2:                     # Namespace, can be anything.
+               py-version: [2, 7]      # Constraint to specific interpreter version
+               path: /opt/2019.2/salt  # Main Salt installation directory.
+               auto_detect: True       # Auto detect dependencies
+               py_bin: /usr/bin/python2.7 # Python binary path to auto detect dependencies
+               dependencies:           # List of dependencies and their installation paths
+                 jinja2: /opt/jinja2
+
+If a dependency is defined in the ``dependecies`` list ``ssh_ext_alternatives`` will use
+this dependency, instead of the path that ``auto_detect`` finds. For example, if you define
+``/opt/jinja2`` under your ``dependencies`` for jinja2, it will not try to autodetect the
+file path to the jinja2 module, and will favor ``/opt/jinja2``.

salt/utils/thin.py

@@ -181,6 +181,62 @@ def gte():
     return salt.utils.json.dumps(tops, ensure_ascii=False)
 
 
+def get_tops_python(py_ver, exclude=None):
+    """
+    Get top directories for the ssh_ext_alternatives dependencies
+    automatically for the given python version. This allows
+    the user to add the dependency paths automatically.
+
+    :param py_ver:
+        python binary to use to detect binaries
+
+    :param exclude:
+        list of modules not to auto detect
+    """
+    files = {}
+    for mod in [
+        "jinja2",
+        "yaml",
+        "tornado",
+        "msgpack",
+        "certifi",
+        "singledispatch",
+        "concurrent",
+        "singledispatch_helpers",
+        "ssl_match_hostname",
+        "markupsafe",
+        "backports_abc",
+    ]:
+        if exclude and mod in exclude:
+            continue
+
+        if not salt.utils.path.which(py_ver):
+            log.error(
+                "{} does not exist. Could not auto detect dependencies".format(py_ver)
+            )
+            return {}
+        py_shell_cmd = "{0} -c 'import {1}; print({1}.__file__)'".format(py_ver, mod)
+        cmd = subprocess.Popen(py_shell_cmd, stdout=subprocess.PIPE, shell=True)
+        stdout, _ = cmd.communicate()
+        mod_file = os.path.abspath(salt.utils.data.decode(stdout).rstrip("\n"))
+
+        if not stdout or not os.path.exists(mod_file):
+            log.error(
+                "Could not auto detect file location for module {} for python version {}".format(
+                    mod, py_ver
+                )
+            )
+            continue
+
+        if os.path.basename(mod_file).split(".")[0] == "__init__":
+            mod_file = os.path.dirname(mod_file)
+        else:
+            mod_file = mod_file.replace("pyc", "py")
+
+        files[mod] = mod_file
+    return files
+
+
 def get_ext_tops(config):
     """
     Get top directories for the dependencies, based on external configuration.
@@ -363,6 +419,76 @@ def _get_thintar_prefix(tarname):
     return tmp_tarname
 
 
+def _pack_alternative(extended_cfg, digest_collector, tfp):
+    # Pack alternative data
+    config = copy.deepcopy(extended_cfg)
+    # Check if auto_detect is enabled and update dependencies
+    for ns, cfg in _six.iteritems(config):
+        if cfg.get("auto_detect"):
+            py_ver = "python" + str(cfg.get("py-version", [""])[0])
+            if cfg.get("py_bin"):
+                py_ver = cfg["py_bin"]
+
+            exclude = []
+            # get any manually set deps
+            deps = config[ns].get("dependencies")
+            if deps:
+                for dep in deps.keys():
+                    exclude.append(dep)
+            else:
+                config[ns]["dependencies"] = {}
+
+            # get auto deps
+            auto_deps = get_tops_python(py_ver, exclude=exclude)
+            for dep in auto_deps:
+                config[ns]["dependencies"][dep] = auto_deps[dep]
+
+    for ns, cfg in _six.iteritems(get_ext_tops(config)):
+        tops = [cfg.get("path")] + cfg.get("dependencies")
+        py_ver_major, py_ver_minor = cfg.get("py-version")
+
+        for top in tops:
+            top = os.path.normpath(top)
+            base, top_dirname = os.path.basename(top), os.path.dirname(top)
+            os.chdir(top_dirname)
+            site_pkg_dir = (
+                _is_shareable(base) and "pyall" or "py{0}".format(py_ver_major)
+            )
+            log.debug(
+                'Packing alternative "%s" to "%s/%s" destination',
+                base,
+                ns,
+                site_pkg_dir,
+            )
+            if not os.path.exists(top):
+                log.error(
+                    "File path {} does not exist. Unable to add to salt-ssh thin".format(
+                        top
+                    )
+                )
+                continue
+            if not os.path.isdir(top):
+                # top is a single file module
+                if os.path.exists(os.path.join(top_dirname, base)):
+                    tfp.add(base, arcname=os.path.join(ns, site_pkg_dir, base))
+                continue
+            for root, dirs, files in salt.utils.path.os_walk(base, followlinks=True):
+                for name in files:
+                    if not name.endswith((".pyc", ".pyo")):
+                        digest_collector.add(os.path.join(root, name))
+                        arcname = os.path.join(ns, site_pkg_dir, root, name)
+                        if hasattr(tfp, "getinfo"):
+                            try:
+                                tfp.getinfo(os.path.join(site_pkg_dir, root, name))
+                                arcname = None
+                            except KeyError:
+                                log.debug(
+                                    'ZIP: Unable to add "%s" with "getinfo"', arcname
+                                )
+                        if arcname:
+                            tfp.add(os.path.join(root, name), arcname=arcname)
+
+
 def gen_thin(
     cachedir,
     extra_mods="",
@@ -597,44 +723,9 @@ def gen_thin(
                 shutil.rmtree(tempdir)
                 tempdir = None
 
-    # Pack alternative data
     if extended_cfg:
         log.debug("Packing libraries based on alternative Salt versions")
-    for ns, cfg in _six.iteritems(get_ext_tops(extended_cfg)):
-        tops = [cfg.get("path")] + cfg.get("dependencies")
-        py_ver_major, py_ver_minor = cfg.get("py-version")
-        for top in tops:
-            base, top_dirname = os.path.basename(top), os.path.dirname(top)
-            os.chdir(top_dirname)
-            site_pkg_dir = (
-                _is_shareable(base) and "pyall" or "py{0}".format(py_ver_major)
-            )
-            log.debug(
-                'Packing alternative "%s" to "%s/%s" destination',
-                base,
-                ns,
-                site_pkg_dir,
-            )
-            if not os.path.isdir(top):
-                # top is a single file module
-                if os.path.exists(os.path.join(top_dirname, base)):
-                    tfp.add(base, arcname=os.path.join(ns, site_pkg_dir, base))
-                continue
-            for root, dirs, files in salt.utils.path.os_walk(base, followlinks=True):
-                for name in files:
-                    if not name.endswith((".pyc", ".pyo")):
-                        digest_collector.add(os.path.join(root, name))
-                        arcname = os.path.join(ns, site_pkg_dir, root, name)
-                        if hasattr(tfp, "getinfo"):
-                            try:
-                                tfp.getinfo(os.path.join(site_pkg_dir, root, name))
-                                arcname = None
-                            except KeyError:
-                                log.debug(
-                                    'ZIP: Unable to add "%s" with "getinfo"', arcname
-                                )
-                        if arcname:
-                            tfp.add(os.path.join(root, name), arcname=arcname)
+        _pack_alternative(extended_cfg, digest_collector, tfp)
 
     os.chdir(thindir)
     with salt.utils.files.fopen(thinver, "w+") as fp_: