-
Notifications
You must be signed in to change notification settings - Fork 152
[WIP] - Libssh2 based clients #86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9f3df25
to
7304b6f
Compare
6c34e3b
to
5d3b621
Compare
…t use hardcoded name. Improved logic of finished function and get_output
… Added programmatic public key authentication to libssh2 client. Updated libssh2 client. Added libssh2 client tests and parallel client tests based on libssh2.
…sh server start to wait for port to be opened
Updated embedded server. Updated setup.py Updated libssh2 client tests.
Updated ssh2 client for new API. Updated ssh2 client tests.
Updated ssh2 client. Updated setup.py
…tion using openmp.
Updated native functions
… be sub-class of it. Updated embedded openssh server to take listen IP parameter and be able to handle starting multiple servers at same time. Updated ssh2 client to handle encoding, sudo, shell, allow agent and timeout parameters and to handle authentication, session and connection errors on session startup/auth. Added SessionError exception. Updated libssh2 parallel client integration tests to cover all currently supported functionality of the libssh2 based client.
PEP8 and cleanups for ssh2 parallel and single clients. Re-enabled travis tests for all clients.
Cleaned up ssh2 client. Utils and test updates.
Updated native select function, parallel ssh2 client test cleanups.
Updated base pssh module and paramiko parallel client. Enabled parallel SFTP operation tests for ssh2 parallel client. Added SFTP exceptions for ssh2 clients. Updated native functions. Updated embedded openssh server.
Updated readme. Added changelog.
…orm compatibility.
Codecov Report
@@ Coverage Diff @@
## master #86 +/- ##
==========================================
+ Coverage 89.63% 92.41% +2.77%
==========================================
Files 8 11 +3
Lines 444 844 +400
==========================================
+ Hits 398 780 +382
- Misses 46 64 +18
Continue to review full report at Codecov.
|
Updated travis, appveyor cfgs. Updated readme. Updated documentation. Updated sftp tests.
… by themselves. Updated changelog.
… and tests. Updated changelog, documentation.
1814e10
to
8e65b6e
Compare
Updated appveyor cfg and upload script. Fixed output line parsing bug, added test.
18 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a WIP pull request for new libssh2 based clients, created to give visibility on this work in progress.
TODO:
SSHClient
ParallelSSHClient
using libssh2 based clientMotivation
Motivation for creating new libssh2 clients as opposed to continuing with paramiko:
Performance.
Libssh2 has been tested to be several orders of magnitude faster than paramiko and other SSH library alternatives, ranging from simple connection and authentication overhead to large data transfer performance.
Stability.
Paramiko often suffers from memory leaks and regressions, ranging from compatibility issues to API and stability/performance regressions, the latter particularly evident with paramiko version
2
. Libssh2 OTOH is a well tested native code base, used in many other applications.Asynchronous/non-blocking support.
Paramiko does not support non-blocking operations. It also makes use of threads and suffers from race conditions in several areas which
parallel-ssh
has had to work around. Its use inparallel-ssh
therefore relies on monkey patching the python standard library via gevent in order to make it non-blocking.This has several drawbacks, most importantly that all other python libraries also have to use the monkey patched standard library and may not work correctly as a result.
Libssh2 has native support for non-blocking mode which allows for a natively non-blocking client to be written, sans monkey patching, directly using the gevent API. It is also thread safe and can be further scaled beyond what a single event loop handling thread can handle.
Native code extension dependency
Why libssh2, why depend on native code extensions?
There are not many options for SSH libraries in python. Paramiko is the de facto standard, mostly due to said lack of options. Paramiko itself depends on other native code extensions for the heavy lifting, particularly around cryptography.
Alternatives to paramiko like asyncssh are python 3 only and also make use of third party native code extensions, while at the same time using an incompatible license. Re-rolling yet another SSH library is not considered as it is (a) too time consuming and (b) would be re-inventing the wheel.
In other words, there is no option which does not rely in some way on native code extensions. The argument for not having to depend on native code extensions is therefore moot. So why not use an existing, well tested, well performing native code library directly via python bindings.
Options for that are similarly limited and it comes down to libssh2 vs libssh (no relation). Libssh2 was chosen for its better performance and greater number of features.
Features
Features of the new clients:
Missing Features
Features in paramiko not present in libssh2 at this time of writing:
The intention is to provide both the existing paramiko and new libssh2 based clients (both parallel and single clients), allowing users to choose which to use depending on requirements.
Scope
New Python wrapper for libssh2 has been written from scratch as ssh2-python and available as a stand alone library. It implements vast majority of the libssh2 API including all server side features. A new from scratch library using Cython including a C-API layer was preferred to having to extensively modify pylibssh2 (hand crafted C), along with some issues found in it.
ssh2-python is mostly feature complete at this stage with support up to latest libssh2 release and includes binary wheels as well as system packages.
Unfortunately, python bindings for libssh2 have been un-maintained for years now and the last release (from 2011), is missing key features since added by third parties. The original project has therefor been forked at ParallelSSH/pylibssh2 and several changes and additions made, including merging of PRs, to be used byparallel-ssh
. Luckily have experience with writing python bindings of C code, have written some extension libraries as well - see NodeTrie.This fork is intended for use byparallel-ssh
and has had its package name changed so as not to conflict with system installed versions of the original library. It will most likely be shipped as part ofparallel-ssh
, possibly in addition to being published at PyPi under a new name. There is no intention to maintain or support the forked project for use by other libraries, though any contributions to it will be reviewed on a case by case basis.Implications to users and package maintainers
ParallelSSHClient
will have a slightly different end-user API. Whether the existing client will change, or a new one made, is still TBD. Never the less, any breaking API changes will only come in a new major version (2.x.x
).parallel-ssh
(likely), package architecture will change fromnoarch
tox86_64
. Binary wheels will be provided.Contributions
Contributions are most welcome and highly encouraged!
Feel free to contribute either on this here branch or at ssh2-python.
The libssh2 clients in this branch are currently usable for testing purposes only.