Worker.Tree: fix selection of remote Python executable (#337)

The Python interpeter on gateways must be of the same major version than
on the root node for the tree mode to work. Use the same executable name
(eg. python, python2, python3) by default.

This is due to changes made to the pickle protocol:

https://docs.python.org/3/library/pickle.html#data-stream-format

Also slashes (/) are not supported in module name (-m), so use dot (.)
instead.

Finally, check the following environment variable:

    CLUSTERSHELL_GW_PYTHON_EXECUTABLE

that can be used to override the default selection, giving users an
option to provide an absolute path to the remote python executable.

Change-Id: If8af8abb202ec13fb7db063859f65283b4923d64

doc/sphinx/tools/clush.rst

@@ -270,6 +270,21 @@ gateway. That is, if the *fanout* is **16**, each gateway will initate up to
    command on the cluster. In tree mode, please note that in that case, each
    gateway will be able to run a command at the same time.
 
+Remote Python executable
+""""""""""""""""""""""""
+
+You must use the same major version of Python on the gateways and the root
+node. By default, the same python executable name than the one used on the
+root node will be used to launch the gateways, that is, `python` or `python3`
+(using relative path for added flexibility). You may override the selection
+of the remote Python interpreter by defining the following environment
+variable::
+
+    $ export CLUSTERSHELL_GW_PYTHON_EXECUTABLE=/path/to/python3
+
+.. note:: It is highly recommended to have the same Python interpeter
+   installed on all gateways and the root node.
+
 Debugging Tree mode
 """""""""""""""""""
 

lib/ClusterShell/Worker/Tree.py

@@ -27,6 +27,7 @@
 import logging
 import os
 from os.path import basename, dirname, isfile, normpath
+import sys
 import tarfile
 import tempfile
 
@@ -175,7 +176,15 @@ def __init__(self, nodes, handler, timeout, **kwargs):
             envval = os.getenv(envname)
             if envval:
                 invoke_gw_args.append("%s=%s" % (envname, envval))
-        invoke_gw_args.append("python -m ClusterShell/Gateway -Bu")
+
+        # It is critical to launch a remote Python executable with the same
+        # major version (ie. python or python3) as we use the (default) pickle
+        # protocol and for example, version 3+ (Python 3 with bytes
+        # support) cannot be unpickled by Python 2.
+        python_executable = os.getenv('CLUSTERSHELL_GW_PYTHON_EXECUTABLE',
+                                      basename(sys.executable or 'python'))
+        invoke_gw_args.append(python_executable)
+        invoke_gw_args.extend(['-m', 'ClusterShell.Gateway', '-Bu'])
         self.invoke_gateway = ' '.join(invoke_gw_args)
 
         self.topology = kwargs.get('topology')