added thread safety and ability to add new hosts

README.rst

@@ -73,15 +73,17 @@ Use ``-e`` only to install in editable mode
 If you encounter some import errors try installing from requirements.txt file:
 ``pip install requirements.txt``
 
+Warning
+-------
+
+There has been a recent mayor change in modules API betweeen versions 0.4.2
+and 0.5.0. Most methods of the connection classes have been moved a level
+deeper. See `migration from 0.4.x to 0.5.x <https://ssh-utilities.readthedocs.io/en/latest/migration.html>`_
+for details how to port to newer version
+
 API and documentation
 ---------------------
 
-.. warning:: 
-    There has been a recent mayor change in modules API betweeen versions 0.4.2
-    and 0.5.0. Most methods of the connection classes have been moved a level
-    deeper. See `migration from 0.4.x to 0.5.x <https://ssh-utilities.readthedocs.io/en/latest/migration.html>`_
-    for details how to port to newer version
-
 It is recommended that you have configured **rsa** keys with config file according
 to `openssh standard <https://www.ssh.com/ssh/config/>`_. For easy quickstart guide
 you can look at: https://www.cyberciti.biz/faq/create-ssh-config-file-on-linux-unix/
@@ -132,7 +134,8 @@ for more detailed usage examples please refer to
 `documnetation <https://ssh-utilities.readthedocs.io/en/latest/>`_
 
 ``Connection`` factory supports dict-like indexing by values that are in
-your **~/.ssh/config** file
+your **~/.ssh/config** file. It can be made thread safe by passing
+``thread_safe=True`` argument to the constructor
 
 .. code-block:: python
 
@@ -146,15 +149,15 @@ support than dict-like indexing
 .. code-block:: python
 
     >>> from ssh_utilities import Connection
-    >>> Connection.get(<server_name>)
+    >>> Connection.get(<server_name>, <local>, <quiet>, <thread_safe>)
     >>> <ssh_utilities.ssh_utils.SSHConnection at 0x7efedff4fb38>
 
 Class can be also used as a context manager.
 
 .. code-block:: python
 
     >>> from ssh_utilities import Connection
-    >>> with Connection(<server_name>) as conn:
+    >>> with Connection(<server_name>, <local>, <quiet>, <thread_safe>) as conn:
     >>>     conn.something(...)
 
 Connection can also be initialized from appropriately formated string.
@@ -173,7 +176,7 @@ customization is required, use open method, this also allows use of passwords
 
     >>> from ssh_utilities import Connection
     >>> conn = Connection.open(<ssh_username>, <ssh_server>, <ssh_key_file>, <server_name>,
-                               <share_connection>):
+                               <thread_safe>):
 
 Module API also exposes powerfull SSHPath object with identical API as
 ``pathlib.Path`` only this one works for remote files. It must be always tied to
@@ -211,5 +214,4 @@ LGPL-2.1
 
 TODO
 ----
-- make connection object thread-safe
 - implement wrapper for pool of connections

docs/source/usage_conn.rst

@@ -5,29 +5,40 @@ Instantiating connection
 ------------------------
 
 ``Connection`` factory supports dict-like indexing by values that are in
-your **~/.ssh/config** file
+your **~/.ssh/config** file. It can be made thread safe by passing
+``thread_safe=True`` argument to the constructor. Your ``~/.ssh.config`` file
+is parsed upon module import and the ``Connection`` factory is indexable by
+values in this file.
 
 .. code-block:: python
 
     >>> from ssh_utilities import Connection
     >>> Connection[<server_name>]
     >>> <ssh_utilities.ssh_utils.SSHConnection at 0x7efedff4fb38>
 
+More hosts can be simply added.
+
+.. code-block:: python
+
+    >>> from ssh_utilities import Connection
+    >>> Connection.add_hosts({"user": <some_user>, "hostname": <my_ssh_server>,
+                              "identityfile": <path_to_my_identity_file>})
+
 There is also a specific get method which is safer and with better typing
 support than dict-like indexing
 
 .. code-block:: python
 
     >>> from ssh_utilities import Connection
-    >>> Connection.get(<server_name>)
+    >>> Connection.get(<server_name>, <local>, <quiet>, <thread_safe>)
     >>> <ssh_utilities.ssh_utils.SSHConnection at 0x7efedff4fb38>
 
 Class can be also used as a context manager.
 
 .. code-block:: python
 
     >>> from ssh_utilities import Connection
-    >>> with Connection(<server_name>) as conn:
+    >>> with Connection(<server_name>, <local>, <quiet>, <thread_safe>) as conn:
     >>>     conn.something(...)
 
 Connection can also be initialized from appropriately formated string.
@@ -46,7 +57,7 @@ customization is required, use open method, this also allows use of passwords
 
     >>> from ssh_utilities import Connection
     >>> conn = Connection.open(<ssh_username>, <ssh_server>, <ssh_key_file>, <server_name>,
-                               <share_connection>)
+                               <thread_safe>)
     >>>
 
 Using connection - subprocess

ssh_utilities/__init__.py

@@ -22,8 +22,9 @@
 from .remote.path import SSHPath
 from .remote import SSHConnection, PIPE, STDOUT, DEVNULL
 from .constants import GET, PUT
+from .utils import config_parser
 
 __all__ = ["SSHConnection", "Connection", "LocalConnection", "SSHPath", "PIPE",
-           "STDOUT", "DEVNULL", "GET", "PUT"]
+           "STDOUT", "DEVNULL", "GET", "PUT", "config_parser"]
 
 logging.getLogger(__name__)

ssh_utilities/base.py

@@ -265,7 +265,8 @@ def _path2str(path: Optional["_SPATH"]) -> str:
 
     @staticmethod
     def to_str(connection_name: str, host_name: str, address: Optional[str],
-               user_name: str, ssh_key: Optional[Union[Path, str]]) -> str:
+               user_name: str, ssh_key: Optional[Union[Path, str]],
+               thread_safe: bool) -> str:
         """Aims to ease persistance, returns string representation of instance.
 
         With this method all data needed to initialize class are saved to sting
@@ -284,6 +285,10 @@ def to_str(connection_name: str, host_name: str, address: Optional[str],
             server login name
         ssh_key : Optional[Union[Path, str]]
             file with public key, pass none for LocalConnection
+        thread_safe: bool
+            make connection object thread safe so it can be safely accessed
+            from  any number of threads, it is disabled by default to avoid
+            performance  penalty of threading locks
 
         Returns
         -------
@@ -301,7 +306,8 @@ def to_str(connection_name: str, host_name: str, address: Optional[str],
         return (f"<{connection_name}:{host_name}>("
                 f"user_name:{user_name} | "
                 f"rsa_key:{key_path} | "
-                f"address:{address})")
+                f"address:{address} | "
+                f"threadsafe:{thread_safe})")
 
     def __del__(self):
         self.close(quiet=True)

ssh_utilities/connection.py

@@ -21,6 +21,11 @@
 if TYPE_CHECKING:
     from pathlib import Path
 
+    from typing_extensions import TypedDict
+
+    _HOSTS = TypedDict("_HOSTS", {"user": str, "hostname": str,
+                                  "identityfile": Union[str, List[str]]})
+
 __all__ = ["Connection"]
 
 logging.getLogger(__name__)
@@ -37,7 +42,6 @@ class _ConnectionMeta(type):
     The inheriting classes can be indexed by keys in ~/.ssh/config file
     """
 
-    SHARE_CONNECTION: int = 10
     available_hosts: Dict
 
     def __new__(cls, classname, bases, dictionary: dict):
@@ -54,7 +58,7 @@ def __new__(cls, classname, bases, dictionary: dict):
         return type.__new__(cls, classname, bases, dictionary)
 
     def __getitem__(cls, key: str) -> Union[SSHConnection, LocalConnection]:
-        return cls.get(key, local=False, quiet=False)
+        return cls.get(key, local=False, quiet=False, thread_safe=False)
 
     def get(cls, *args, **kwargs) -> Union[SSHConnection, LocalConnection]:
         """Overriden in class that inherits this metaclass."""
@@ -112,12 +116,13 @@ class Connection(metaclass=_ConnectionMeta):
 
     >>> from ssh_utilities import Connection
     >>> conn = Connection.open(<ssh_username>, <ssh_server>, <ssh_key_file>,
-                               <server_name>, <share_connection>):
+                               <server_name>, <thread_safe>):
     """
 
     def __init__(self, ssh_server: str, local: bool = False,
-                 quiet: bool = False) -> None:
-        self._connection = self.get(ssh_server, local=local, quiet=quiet)
+                 quiet: bool = False, thread_safe: bool = False) -> None:
+        self._connection = self.get(ssh_server, local=local, quiet=quiet,
+                                    thread_safe=thread_safe)
 
     def __enter__(self) -> Union[SSHConnection, LocalConnection]:
         return self._connection
@@ -148,24 +153,25 @@ def get_available_hosts(cls) -> List[str]:
 
     @overload
     @classmethod
-    def get(cls, ssh_server: str, local: Literal[False], quiet: bool
-            ) -> SSHConnection:
+    def get(cls, ssh_server: str, local: Literal[False], quiet: bool,
+            thread_safe: bool) -> SSHConnection:
         ...
 
     @overload
     @classmethod
     def get(cls, ssh_server: str, local: Literal[True], quiet: bool,
-            ) -> LocalConnection:
+            thread_safe: bool) -> LocalConnection:
         ...
 
     @overload
     @classmethod
     def get(cls, ssh_server: str, local: bool, quiet: bool,
-            ) -> Union[SSHConnection, LocalConnection]:
+            thread_safe: bool) -> Union[SSHConnection, LocalConnection]:
         ...
 
     @classmethod
-    def get(cls, ssh_server: str, local: bool = False, quiet: bool = False):
+    def get(cls, ssh_server: str, local: bool = False, quiet: bool = False,
+            thread_safe: bool = False):
         """Get Connection based on one of names defined in .ssh/config file.
 
         If name of local PC is passed initilize LocalConnection
@@ -178,6 +184,10 @@ def get(cls, ssh_server: str, local: bool = False, quiet: bool = False):
             if True return emulated connection to loacl host
         quiet: bool
             If True suppress login messages
+        thread_safe: bool
+            make connection object thread safe so it can be safely accessed
+            from  any number of threads, it is disabled by default to avoid
+            performance  penalty of threading locks
 
         Raises
         ------
@@ -203,13 +213,39 @@ def get(cls, ssh_server: str, local: bool = False, quiet: bool = False):
                 return cls.open(credentials["user"], credentials["hostname"],
                                 credentials["identityfile"][0],
                                 server_name=ssh_server, quiet=quiet,
-                                share_connection=cls.SHARE_CONNECTION)
+                                thread_safe=thread_safe)
             except KeyError as e:
                 raise KeyError(f"{RED}missing key in config dictionary for "
                                f"{ssh_server}: {R}{e}")
 
     get_connection = get
 
+    @classmethod
+    def add_hosts(cls, hosts: Union["_HOSTS", List["_HOSTS"]]):
+        """add or override availbale host read fron ssh config file.
+
+        You can use supplied config parser to parse some externaf ssh config
+        file.
+
+        Parameters
+        ----------
+        hosts : Union[_HOSTS, List[_HOSTS]]
+            dictionary or a list of dictionaries containing keys: `user`,
+            `hostname` and `identityfile`
+
+        See also
+        --------
+        :func: ssh_utilities.config_parser
+        """
+        if not isinstance(hosts, list):
+            hosts = [hosts]
+
+        for h in hosts:
+            if not isinstance(h["identityfile"], list):
+                h["identityfile"] = [h["identityfile"]]
+
+        cls.available_hosts.update({h: h["hostname"] for h in hosts})
+
     @classmethod
     def from_str(cls, string: str, quiet: bool = False
                  ) -> Union[SSHConnection, LocalConnection]:
@@ -239,41 +275,20 @@ def from_str(cls, string: str, quiet: bool = False
         try:
             server_name = re.findall(r"<\S*:(\S*)>", string)[0]
             user_name, ssh_key, address = re.findall(
-                r"\(user_name:(\S*) \| rsa_key:(\S*) \| address:(\S*)\)",
-                string)[0]
+                r"\(user_name:(\S*) \| rsa_key:(\S*) \| address:(\S*) \| "
+                r"threadsafe:(\S*)\)", string)[0]
         except IndexError:
             raise ValueError("String is not formated correctly")
 
         return cls.open(user_name, address, ssh_key, server_name, quiet=quiet)
 
-    @classmethod
-    def set_shared(cls, number_of_shared: Union[int, bool]):
-        """Set how many instancesd can share the same connection to server.
-
-        Parameters
-        ----------
-        number_of_shared: Union[int, bool]
-            if int number of shared instances is set to that number
-            if False number of shared instances is set to 0
-            if True number of shared instances is set to 10
-
-        Warnings
-        --------
-        This is not implemented yet!
-        """
-        if number_of_shared is True:
-            number_of_shared = 10
-        elif number_of_shared is False:
-            number_of_shared = 0
-        cls.SHARE_CONNECTION = number_of_shared
-
     @overload
     @staticmethod
     def open(ssh_username: str, ssh_server: None = None,
              ssh_key_file: Optional[Union[str, "Path"]] = None,
              ssh_password: Optional[str] = None,
              server_name: Optional[str] = None, quiet: bool = False,
-             share_connection: int = 10) -> LocalConnection:
+             thread_safe: bool = False) -> LocalConnection:
         ...
 
     @overload
@@ -282,15 +297,15 @@ def open(ssh_username: str, ssh_server: str,
              ssh_key_file: Optional[Union[str, "Path"]] = None,
              ssh_password: Optional[str] = None,
              server_name: Optional[str] = None, quiet: bool = False,
-             share_connection: int = 10) -> SSHConnection:
+             thread_safe: bool = False) -> SSHConnection:
         ...
 
     @staticmethod
     def open(ssh_username: str, ssh_server: Optional[str] = "",
              ssh_key_file: Optional[Union[str, "Path"]] = None,
              ssh_password: Optional[str] = None,
              server_name: Optional[str] = None, quiet: bool = False,
-             share_connection: int = 10):
+             thread_safe: bool = False):
         """Initialize SSH or local connection.
 
         Local connection is only a wrapper around os and shutil module methods
@@ -313,9 +328,10 @@ def open(ssh_username: str, ssh_server: Optional[str] = "",
             default than it will be replaced with address.
         quiet: bool
             If True suppress login messages
-        share_connection: int
-            share connection between different instances of class, number says
-            how many instances can share the same connection
+        thread_safe: bool
+            make connection object thread safe so it can be safely accessed
+            from  any number of threads, it is disabled by default to avoid
+            performance  penalty of threading locks
 
         Warnings
         --------
@@ -331,14 +347,14 @@ def open(ssh_username: str, ssh_server: Optional[str] = "",
                 c = SSHConnection(ssh_server, ssh_username,
                                   rsa_key_file=ssh_key_file, line_rewrite=True,
                                   server_name=server_name, quiet=quiet,
-                                  share_connection=share_connection)
+                                  thread_safe=thread_safe)
             else:
                 if not ssh_password:
                     ssh_password = getpass.getpass(prompt="Enter password: ")
 
                 c = SSHConnection(ssh_server, ssh_username,
                                   password=ssh_password, line_rewrite=True,
                                   server_name=server_name, quiet=quiet,
-                                  share_connection=share_connection)
+                                  thread_safe=thread_safe)
 
             return c