bug: Add options to SFTP (#2843)

Noticed authentication errors when connected to a non localhost SFTP. It
errored out when looking for ssh keys.

This gives us the option to not look for those. Which is correct if we
are giving it user/password.

CHANGELOG.md

@@ -1,4 +1,4 @@
-## 0.13.1-dev2
+## 0.13.1-dev3
 
 ### Enhancements
 
@@ -9,6 +9,7 @@
 ### Fixes
 
 * **Fix `partition_html()` swallowing some paragraphs**. The `partition_html()` only considers elements with limited depth to avoid becoming the text representation of a giant div. This fix increases the limit value.
+* **Fix SFTP** Adds flag options to SFTP connector on whether to use ssh keys / agent, with flag values defaulting to False. This is to prevent looking for ssh files when using username and password. Currently, username and password are required, making that always the case.
 
 ## 0.13.0
 

unstructured/__version__.py

@@ -1 +1 @@
-__version__ = "0.13.1-dev2"  # pragma: no cover
+__version__ = "0.13.1-dev3"  # pragma: no cover

unstructured/ingest/cli/cmds/fsspec/sftp.py

@@ -29,6 +29,22 @@ def get_cli_options() -> t.List[click.Option]:
                 type=str,
                 help="Password for sftp connection",
             ),
+            click.Option(
+                ["--look-for-keys"],
+                required=False,
+                default=False,
+                is_flag=True,
+                type=bool,
+                help="Whether to search for private key files in ~/.ssh/",
+            ),
+            click.Option(
+                ["--allow-agent"],
+                required=False,
+                default=False,
+                is_flag=True,
+                type=bool,
+                help="Whether to connect to the SSH agent.",
+            ),
         ]
         return options
 

unstructured/ingest/connector/fsspec/sftp.py

@@ -22,6 +22,8 @@ class SftpAccessConfig(AccessConfig):
     password: str = enhanced_field(sensitive=True)
     host: str = ""
     port: int = 22
+    look_for_keys: bool = False
+    allow_agent: bool = False
 
 
 @dataclass