Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] - Libssh2 based clients #86

merged 50 commits into from Oct 14, 2017

[WIP] - Libssh2 based clients #86

merged 50 commits into from Oct 14, 2017


Copy link

@pkittenis pkittenis commented Jul 19, 2017

This is a WIP pull request for new libssh2 based clients, created to give visibility on this work in progress.


  • Usable, asynchronous libssh2 based SSHClient
  • Libssh2 client tests - integration tests based on openssh server
  • Programmatic private key authentication
  • SSH agent authentication
  • Standard identity files attempted if available
  • ParallelSSHClient using libssh2 based client
  • Port parallel client tests to libssh2 client (partly done)
  • Raise proper exceptions on errors from libssh2 (libssh2 bindings)
  • Update libssh2 bindings for new style classes
  • Implement libssh2 bindings for all of the libssh2 API
  • Figure out correct way to poll if remote command has finished executing
  • Implement TCP forwarding
  • Implement proxy connections via TCP forwarding
  • Implement connection timeout
  • Implement SFTP client functions
  • Refactor parallel clients for code reuse
  • Finalise public API for libssh2 clients
  • Packaging, wheel building etc


Motivation for creating new libssh2 clients as opposed to continuing with paramiko:

  1. Performance.
    Libssh2 has been tested to be several orders of magnitude faster than paramiko and other SSH library alternatives, ranging from simple connection and authentication overhead to large data transfer performance.

  2. Stability.
    Paramiko often suffers from memory leaks and regressions, ranging from compatibility issues to API and stability/performance regressions, the latter particularly evident with paramiko version 2. Libssh2 OTOH is a well tested native code base, used in many other applications.

  3. Asynchronous/non-blocking support.
    Paramiko does not support non-blocking operations. It also makes use of threads and suffers from race conditions in several areas which parallel-ssh has had to work around. Its use in parallel-ssh therefore relies on monkey patching the python standard library via gevent in order to make it non-blocking.
    This has several drawbacks, most importantly that all other python libraries also have to use the monkey patched standard library and may not work correctly as a result.
    Libssh2 has native support for non-blocking mode which allows for a natively non-blocking client to be written, sans monkey patching, directly using the gevent API. It is also thread safe and can be further scaled beyond what a single event loop handling thread can handle.

Native code extension dependency

  • Why libssh2, why depend on native code extensions?

  • There are not many options for SSH libraries in python. Paramiko is the de facto standard, mostly due to said lack of options. Paramiko itself depends on other native code extensions for the heavy lifting, particularly around cryptography.

    Alternatives to paramiko like asyncssh are python 3 only and also make use of third party native code extensions, while at the same time using an incompatible license. Re-rolling yet another SSH library is not considered as it is (a) too time consuming and (b) would be re-inventing the wheel.

    In other words, there is no option which does not rely in some way on native code extensions. The argument for not having to depend on native code extensions is therefore moot. So why not use an existing, well tested, well performing native code library directly via python bindings.
    Options for that are similarly limited and it comes down to libssh2 vs libssh (no relation). Libssh2 was chosen for its better performance and greater number of features.


Features of the new clients:

  • Vastly improved performance and better scaling in all areas.
  • Asynchronous by nature - no monkey patching.
  • Use the gevent API directly.
  • Allows for the client to be used within other applications with no side effects.

Missing Features

Features in paramiko not present in libssh2 at this time of writing:

  • SSH agent forwarding - libssh2/#185
  • GSS-API authentication (aka kerberos)

The intention is to provide both the existing paramiko and new libssh2 based clients (both parallel and single clients), allowing users to choose which to use depending on requirements.


New Python wrapper for libssh2 has been written from scratch as ssh2-python and available as a stand alone library. It implements vast majority of the libssh2 API including all server side features. A new from scratch library using Cython including a C-API layer was preferred to having to extensively modify pylibssh2 (hand crafted C), along with some issues found in it.

ssh2-python is mostly feature complete at this stage with support up to latest libssh2 release and includes binary wheels as well as system packages.

Unfortunately, python bindings for libssh2 have been un-maintained for years now and the last release (from 2011), is missing key features since added by third parties. The original project has therefor been forked at ParallelSSH/pylibssh2 and several changes and additions made, including merging of PRs, to be used by parallel-ssh. Luckily have experience with writing python bindings of C code, have written some extension libraries as well - see NodeTrie.

This fork is intended for use by parallel-ssh and has had its package name changed so as not to conflict with system installed versions of the original library. It will most likely be shipped as part of parallel-ssh, possibly in addition to being published at PyPi under a new name. There is no intention to maintain or support the forked project for use by other libraries, though any contributions to it will be reviewed on a case by case basis.

Implications to users and package maintainers

  • The libssh2 based ParallelSSHClient will have a slightly different end-user API. Whether the existing client will change, or a new one made, is still TBD. Never the less, any breaking API changes will only come in a new major version (2.x.x).
  • If libssh2 bindings are shipped as part of parallel-ssh (likely), package architecture will change from noarch to x86_64. Binary wheels will be provided.


Contributions are most welcome and highly encouraged!

Feel free to contribute either on this here branch or at ssh2-python.

The libssh2 clients in this branch are currently usable for testing purposes only.

@pkittenis pkittenis added this to the 2.0.0 milestone Jul 19, 2017
@pkittenis pkittenis force-pushed the libssh2 branch 2 times, most recently from 6c34e3b to 5d3b621 Compare October 4, 2017 23:31
…t use hardcoded name. Improved logic of finished function and get_output
… Added programmatic public key authentication to libssh2 client. Updated libssh2 client. Added libssh2 client tests and parallel client tests based on libssh2.
…sh server start to wait for port to be opened
Updated embedded server.
Updated libssh2 client tests.
Updated ssh2 client for new API.
Updated ssh2 client tests.
… be sub-class of it.

Updated embedded openssh server to take listen IP parameter and be able to handle starting multiple servers at same time.
Updated ssh2 client to handle encoding, sudo, shell, allow agent and timeout parameters and to handle authentication, session and connection errors on session startup/auth.
Added SessionError exception.
Updated libssh2 parallel client integration tests to cover all currently supported functionality of the libssh2 based client.
PEP8 and cleanups for ssh2 parallel and single clients.
Re-enabled travis tests for all clients.
Cleaned up ssh2 client.
Utils and test updates.
Updated native select function, parallel ssh2 client test cleanups.
Updated base pssh module and paramiko parallel client.
Enabled parallel SFTP operation tests for ssh2 parallel client.
Added SFTP exceptions for ssh2 clients.
Updated native functions.
Updated embedded openssh server.
Copy link

codecov bot commented Oct 7, 2017

Codecov Report

Merging #86 into master will increase coverage by 2.77%.
The diff coverage is 92.07%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #86      +/-   ##
+ Coverage   89.63%   92.41%   +2.77%     
  Files           8       11       +3     
  Lines         444      844     +400     
+ Hits          398      780     +382     
- Misses         46       64      +18
Impacted Files Coverage Δ
tests/embedded_server/ 100% <ø> (ø)
pssh/ 100% <100%> (ø) ⬆️
pssh/ 100% <100%> (ø)
pssh/ 100% <100%> (ø) ⬆️
pssh/ 100% <100%> (ø) ⬆️
pssh/ 87.14% <85.18%> (-0.3%) ⬇️
pssh/ 89.52% <89.52%> (ø)
pssh/ 96.2% <90%> (+5.97%) ⬆️
pssh/ 97.19% <97.19%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e82198...b5805e9. Read the comment docs.

@pkittenis pkittenis force-pushed the libssh2 branch 3 times, most recently from 1814e10 to 8e65b6e Compare October 13, 2017 12:41
Updated appveyor cfg and upload script.
Fixed output line parsing bug, added test.
@pkittenis pkittenis merged commit 900c3e2 into master Oct 14, 2017
@pkittenis pkittenis mentioned this pull request Oct 14, 2017
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

None yet

1 participant