Switched everything to use cryptography instead of pyCrypto #394

Merged
merged 64 commits into from Apr 25, 2016

Projects

None yet
@alex
Contributor
alex commented Sep 15, 2014

Motivation:

  • Adds PyPy support
  • Performance improvement
  • OpenSSL and friends are better audited than PyCrypto
  • Easier windows install flow (Cryptography provides statically linked wheels
    on Windows)

This PR is basically complete on the code side, of course it can always use
more review :-)

Tests all pass locally (tested with PyPy!)

Still needs some docs work, and to figure out how to do this with the version
numbers so people's stuff doesn't suddenly get broken.

@alex
Contributor
alex commented Sep 15, 2014

Oh, this also requires cryptography master right now, we need to get a release out before this is usable; that's no problem though.

@alex
Contributor
alex commented Sep 15, 2014

Oh -- this also adds pyasn1 as a dependency, it's a pretty small thing, and could be used to replace some of the existing BER stuff in the future. (We can also remove the ecdsa requirement and just use cryptography as well)

@lndbrg lndbrg and 1 other commented on an outdated diff Sep 15, 2014
paramiko/rsakey.py
:return: new `.RSAKey` private key
"""
- rsa = RSA.generate(bits, os.urandom, progress_func)
- key = RSAKey(vals=(rsa.e, rsa.n))
- key.d = rsa.d
- key.p = rsa.p
- key.q = rsa.q
+ numbers = rsa.generate_private_key(
+ 65537, bits, backend=default_backend()
@lndbrg
lndbrg Sep 15, 2014 Contributor

What does 65537 mean here?

@alex
alex Sep 15, 2014 Contributor

65537 is the public_exponent here, it's generally considered a good default (it's what PyCrypto uses). Would passing it as a keyword-argument make this easier?

@lndbrg
lndbrg Sep 16, 2014 Contributor

Yes. :)

@reaperhulk reaperhulk and 2 others commented on an outdated diff Sep 15, 2014
tests/test_client.py
@@ -287,14 +289,10 @@ def test_6_cleanup(self):
self.tc.close()
del self.tc
- # hrm, sometimes p isn't cleared right away. why is that?
- #st = time.time()
- #while (time.time() - st < 5.0) and (p() is not None):
- # time.sleep(0.1)
-
- # instead of dumbly waiting for the GC to collect, force a collection
- # to see whether the SSHClient object is deallocated correctly
- import gc
+ # force a collection to see whether the SSHClient object is deallocated
+ # correctly
+ gc.collect()
@reaperhulk
reaperhulk Sep 15, 2014

Why 3 calls to collect?

@alex
alex Sep 15, 2014 Contributor

For PyPy :-)

On Mon, Sep 15, 2014 at 5:22 PM, Paul Kehrer notifications@github.com
wrote:

In tests/test_client.py:

@@ -287,14 +289,10 @@ def test_6_cleanup(self):
self.tc.close()
del self.tc

  •    # hrm, sometimes p isn't cleared right away.  why is that?
    
  •    #st = time.time()
    
  •    #while (time.time() - st < 5.0) and (p() is not None):
    

- # time.sleep(0.1)

  •    # instead of dumbly waiting for the GC to collect, force a collection
    
  •    # to see whether the SSHClient object is deallocated correctly
    
  •    import gc
    
  •    # force a collection to see whether the SSHClient object is deallocated
    
  •    # correctly
    
  •    gc.collect()
    

Why 3 calls to collect?


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r17569651.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@lndbrg
lndbrg Sep 16, 2014 Contributor

Might be good to have a comment to explain why we have three of them (for pypy) and why it's needed. :)

@lndbrg
Contributor
lndbrg commented Sep 16, 2014

Some tests seems to be failing on CPython. Is that due to requiring master of cryptography?

If you'd like to add pypy and pypy3 to the .travis.yml that would be good, so we'll get those tested as well.

@lndbrg
Contributor
lndbrg commented Sep 16, 2014

Also, is there any significant speed difference going from PyCrypto to cryptography?

@lndbrg
Contributor
lndbrg commented Sep 16, 2014

Most of it looks good though. :) I will look at it a couple of times more to see if i discover something else.

@reaperhulk

@lndbrg I don't believe any of the core developers have conducted a serious benchmark of pyca/cryptography against pycrypto, but I'd expect them to be broadly similar.

The CPython failures are due to a function required for this PR not being present in the currently released version of pyca/cryptography (0.5.4). We'll be releasing a new version of the library soon to correct this (and add some new features).

@lndbrg
Contributor
lndbrg commented Sep 16, 2014

Would it be okay to ask you to try to transfer, say a 100M file, one with this branch and one with pycrypto? :)

@alex
Contributor
alex commented Sep 16, 2014

I haven't benchmarked it with paramiko, but the last time I benchmarked PyCrypto vs Cryptography (this was for DKIM, which is all RSA) it was many many times faster, 6x IIRC.

If you can show me how to benchmark transfering a 100MB file with paramiko, I'm happy to do so :-)

@lndbrg
Contributor
lndbrg commented Sep 17, 2014

Cool. There is a test called test_sftp_big.py that transfers a 1MB file, running that should probably suffice. :)

@lndbrg
Contributor
lndbrg commented Sep 17, 2014

@reaperhulk when the new version is released and we get a green run in travis i'll have another look. 👍

@lndbrg

This should probably go into tox.ini as well.

@bitprophet
Member

SO EXCITED

@bitprophet
Member

Actual comments, without reading the diffs yet:

  • Re: versions: if this only added pure-Python deps, it would be a minor release (eg we added ecdsa support in 1.13 or so) as it adds no "real" dependency - users able to install paramiko beforehand can still install after.
    • However, unless I'm missing something, Cryptography requires OpenSSL & PyCrypto doesn't (PyCrypto only needs regular Python dev headers and I'm reasonably sure those don't themselves require OpenSSL?)
    • If that's accurate I may just say "ok this is now Paramiko 2.0, but there are no API changes, only dependency changes" - perhaps unusual, but honoring semver nonetheless.
  • Re: ecdsa being subsumed, that's fine, and is also more fodder for making it a 2.0 level change (though not a hard requirement).
  • Re: pyasn1, that's already one of the optional requirements for the merged-but-unreleased Kerberos/GSSAPI support, so this just makes it non-optional.
@bitprophet bitprophet commented on an outdated diff Sep 19, 2014
paramiko/ecdsakey.py
@@ -133,9 +133,7 @@ def generate(curve=curves.NIST256p, progress_func=None):
@param bits: number of bits the generated key should be.
@type bits: int
- @param progress_func: an optional function to call at key points in
- key generation (used by C{pyCrypto.PublicKey}).
- @type progress_func: function
+ @param progress_func: Unused.
@bitprophet
bitprophet Sep 19, 2014 Member

Heh, this junk here is some old epydoc syntax. I just fixed it (& scanned for any more elsewhere) and updated master. You'll probably need to update to account.

@bitprophet bitprophet and 1 other commented on an outdated diff Sep 19, 2014
'ecdsa >= 0.11',
+ 'pyasn1',
@bitprophet
bitprophet Sep 19, 2014 Member

FTR, the Kerberos ticket we merged recently had (in the install docs - since it's optional) a minimum of pyasn1 >= 0.1.7; I'm thinking it may be worth asserting the same here. Discussion encouraged though.

@alex
alex Sep 19, 2014 Contributor

Err, do you mean maxmimum, or did you get the comparator backwards?

On Thu, Sep 18, 2014 at 5:59 PM, Jeff Forcier notifications@github.com
wrote:

In setup.py:

         'ecdsa >= 0.11',
  •        'pyasn1',
    

FTR, the Kerberos ticket we merged recently had a hard minimum of pyasn1
<= 0.1.7; I'm thinking it may be worth asserting the same here.
Discussion encouraged though.


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r17764863.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
bitprophet Sep 19, 2014 Member

You read it before I edited it :D slow down! ;)

@alex alex Merge branch 'master' into switch-to-cryptography
Conflicts:
	paramiko/ecdsakey.py
	tests/test_client.py
848a720
@bitprophet bitprophet and 1 other commented on an outdated diff Sep 19, 2014
sites/www/installing.rst
C extension
-----------
-Unless you are installing from a precompiled source such as a Debian apt
-repository or RedHat RPM, or using :ref:`pypm <pypm>`, you will also need the
-ability to build Python C-based modules from source in order to install
-PyCrypto. Users on **Unix-based platforms** such as Ubuntu or Mac OS X will
@bitprophet
bitprophet Sep 19, 2014 Member

Why was the "precompiled source / packages" bit removed? Are there no plans to distribute Cryptography via OS channels?

@alex
alex Sep 19, 2014 Contributor

It's in some OSes already (Debian), but not enough that I thought it was
worth noting; disagree?

On Thu, Sep 18, 2014 at 6:01 PM, Jeff Forcier notifications@github.com
wrote:

In sites/www/installing.rst:

C extension


-Unless you are installing from a precompiled source such as a Debian apt
-repository or RedHat RPM, or using :ref:pypm <pypm>, you will also need the
-ability to build Python C-based modules from source in order to install
-PyCrypto. Users on Unix-based platforms such as Ubuntu or Mac OS X will

Why was the "precompiled source / packages" bit removed? Are there no
plans to distribute Cryptography via OS channels?


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r17764948.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
bitprophet Sep 19, 2014 Member

Well, Paramiko is presently packaged on a number of distros (who also usually package PyCrypto) and I'd expect this to continue, even if it means maintainers need to start packaging Cryptography too. Either way I think the note is useful for some newbies and doesn't harm anything by existing.

@alex
alex Sep 19, 2014 Contributor

I absolutely think that as maintainers of distro packages update Parmaiko
they'll package cryptography, if it isn't already available, and we'll work
with those maintainers. (We're also in FreeBSD, adn some other weird
distros :-) and work closely with those upstream maintainers).

On Thu, Sep 18, 2014 at 6:19 PM, Jeff Forcier notifications@github.com
wrote:

In sites/www/installing.rst:

C extension


-Unless you are installing from a precompiled source such as a Debian apt
-repository or RedHat RPM, or using :ref:pypm <pypm>, you will also need the
-ability to build Python C-based modules from source in order to install
-PyCrypto. Users on Unix-based platforms such as Ubuntu or Mac OS X will

Well, Paramiko is presently packaged on a number of distros (who also
usually package PyCrypto) and I'd expect this to continue, even if it means
maintainers need to start packaging Cryptography too. Either way I think
the note is useful for some newbies and doesn't harm anything by existing.


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r17765412.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet bitprophet and 1 other commented on an outdated diff Sep 19, 2014
sites/www/installing.rst
-- basically, anything with ``gcc``, ``make`` and so forth) as well as the
-Python development libraries, often named ``python-dev`` or similar.
+Python development libraries, often named ``python-dev`` or similar, and libffi
+development libraries, often named ``libffi-dev``.
@bitprophet
bitprophet Sep 19, 2014 Member

Don't we need an entry here for OpenSSL too?

@alex
alex Sep 19, 2014 Contributor

Good catch.

On Thu, Sep 18, 2014 at 6:02 PM, Jeff Forcier notifications@github.com
wrote:

In sites/www/installing.rst:

-- basically, anything with gcc, make and so forth) as well as the
-Python development libraries, often named python-dev or similar.
+Python development libraries, often named python-dev or similar, and libffi
+development libraries, often named libffi-dev.

Don't we need an entry here for OpenSSL (& its dev headers) too?


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r17764970.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

Added some line notes (see above), otherwise this looks good to me at a glance. I haven't yet installed master Cryptography to test it out though.

@alex
Contributor
alex commented Sep 19, 2014

Pushed a fix for the merge conflict and the docs issue you noted.

On Thu, Sep 18, 2014 at 6:04 PM, Jeff Forcier notifications@github.com
wrote:

Added some line notes (see above), otherwise this looks good to me at a
glance. I haven't yet installed master Cryptography to test it out though.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet bitprophet referenced this pull request in fabric/fabric Sep 19, 2014
Open

Support pypy #1189

@bitprophet bitprophet added this to the 2.0 milestone Sep 19, 2014
@bitprophet
Member

Got it, thanks. Looks good.

I've added this to a 2.0 milestone as per my previous mainline comment re: requiring OpenSSL being a (dependency-level) backwards compatibility break.

For now what I'm planning is to gather up a few more feature tickets (notably #387 which will be a moderate change, albeit one not changing actual API) and then cut 2.0 as the next feature release (so we'd go 1.14, 1.15, 2.0, 2.1, ...). I don't have a strict timeline for this, but I do need to focus on Fabric 2 / Invoke work for a while - and #387 may be a PITA to develop - so it may be a month-ish or more.

@bitprophet
Member

That said, by putting this in the milestone it means next time I go "it's paramiko time!" this will get released, for whatever that's worth :)

@alex
Contributor
alex commented Sep 19, 2014

Cool, we'll aim to have the necessary Cryptography release out by the end
of next week; do you anticipate wanting to release before then (if yes, we
can do a release sooner)

On Thu, Sep 18, 2014 at 6:25 PM, Jeff Forcier notifications@github.com
wrote:

That said, by putting this in the milestone it means next time I go "it's
paramiko time!" this will get released, for whatever that's worth :)


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

Nope, definitely not before then, I have a mountain of work re: Fabric 2 ahead of me. So no time pressure from me on that.

@alex
Contributor
alex commented Sep 19, 2014

Coolio.

On Thu, Sep 18, 2014 at 6:29 PM, Jeff Forcier notifications@github.com
wrote:

Nope, definitely not before then, I have a mountain of work re: Fabric 2
ahead of me. So no time pressure from me on that.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@OOPMan
OOPMan commented Sep 26, 2014

Super excited about this, looking forward to running Fabric using PyPy :-)

@bitprophet
Member

@OOPMan FYI there's a good chance Fabric 1.x will require Paramiko <2.0 due to that dependency change :( I wish Python packaging made it easier for users to override that sort of pinning, but for now I think it'd be better not to screw over people trying to upgrade and suddenly having their C level requirements change under them!

OTOH, Fabric 2 will then be able to preserve the full range of support that Invoke (part of Fabric 2) currently has (Python 2.6, 2.7, 3.2, 3.3, 3.4, PyPy).

@OOPMan
OOPMan commented Sep 26, 2014

@bitprophet Fine by me. I'm quite happy to migrate to a newer version of Fabric if it means I can install it in my PyPy virtualenv :-) Right now Fabric is the only part of my project that isn't PyPy compatible...

@alex
Contributor
alex commented Sep 26, 2014

@bitprophet ideally it would still be possible to use paramiko 2.0+ will
fabric, just installing fabric would get you a paramiko<2.0; does that make
snese?

On Fri, Sep 26, 2014 at 1:38 PM, Jeff Forcier notifications@github.com
wrote:

@OOPMan https://github.com/OOPMan FYI there's a good chance Fabric 1.x
will require Paramiko <2.0 due to that dependency change. OTOH, Fabric 2
will be able to preserve the full range of support that Invoke (part of
Fabric 2) currently has (Python 2.6, 2.7, 3.2, 3.3, 3.4, PyPy).


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

@alex Yes, that's why I was kvetching about how that's just not "easy" with the current packaging setup (to be fair I've no idea how it could be truly easy unless it was some grody shit checking for openssl existence in setup.py).

But because I don't plan on having any truly API backwards compat changes in 2.0, absolutely users should be able to manually upgrade Paramiko after installing Fabric 1.x. Things would still work fine, I just don't want "pip install fabric < 2.0" to suddenly start pulling down the OpenSSL requiring version.

@alex
Contributor
alex commented Sep 27, 2014

Right, perhaps we could get fabric[new-paramiko] to work though?

On Fri, Sep 26, 2014 at 11:15 PM, Jeff Forcier notifications@github.com
wrote:

@alex https://github.com/alex Yes, that's why I was kvetching about how
that's just not "easy" with the current packaging setup.

But because I don't plan on having any truly API backwards compat changes
in 2.0, absolutely users should be able to manually upgrade Paramiko after
installing Fabric 1.x. Things would still work fine, I just don't want "pip
install fabric" to suddenly start pulling down the OpenSSL requiring
version.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

Is that a thing pip/pypi support? (Braces and such.) Otherwise, I could see us exploring additional PyPI entries with slightly distinct requirements data, or whatever.

@alex
Contributor
alex commented Sep 27, 2014

Yeah, the package[extra-name] thing is supported natively, you just
add extras_require={'extra-name': ['list of stuff']} to setup.py; not sure if it'll handle the confliting
versions correctly though.

On Fri, Sep 26, 2014 at 11:17 PM, Jeff Forcier notifications@github.com
wrote:

Is that a thing pip/pypi support? (Braces and such.) Otherwise, I could
see us exploring additional PyPI entries with slightly distinct
requirements data, or whatever.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@alex
Contributor
alex commented Sep 30, 2014

Just rehit the CI since we released the needed version of Cryptography. Tests look good, except for PyPy, which hung. Not sure what's up there, runs very nicely locally:

$ python test.py
................................................................................................................................................ 1s 1s ......... 2s 0s ......... 1s ......... 2s ......... ........ .........  .........  ...
----------------------------------------------------------------------
Ran 145 tests in 30.516s

OK

Can someone with permissions try rerunning that built to see if ti was an issue with the host?

@bitprophet
Member

Did, now it's erroring with this instead: https://travis-ci.org/paramiko/paramiko/jobs/36630793#L414 :(

(The docs error at bottom smells like something else that isn't PyPy's fault.)

@alex
Contributor
alex commented Sep 30, 2014

The doc thing fails on all the builds, looking into the test_6_cleanup
failure.

On Tue, Sep 30, 2014 at 2:13 AM, Jeff Forcier notifications@github.com
wrote:

Did, now it's erroring with this instead:
https://travis-ci.org/paramiko/paramiko/jobs/36630793#L414 :(

(The docs error at bottom smells like something else that isn't PyPy's
fault.)


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@alex
Contributor
alex commented Sep 30, 2014

Not sure what's up with the weakref test; it passes locally 100% of the time, but that test is also skipped on Python3, I wonder if that's the same issue.

@techtonik

I don't mind against having Fabric 32.0 if that helps to push orthogonal features faster.

@bitprophet bitprophet referenced this pull request in fabric/fabric Oct 2, 2014
Closed

Use cryptography.io if available #1119

@bitprophet
Member

@alex Hm that's right, that one was also really odd-behaving on Python 3. If there's a reliable way to skip it on PyPy too I figure we might as well :( Precedent and all that.

@alex
Contributor
alex commented Oct 2, 2014

Done :-(

On Wed, Oct 1, 2014 at 5:05 PM, Jeff Forcier notifications@github.com
wrote:

@alex https://github.com/alex Hm that's right, that one was also really
odd-behaving on Python 3. If there's a reliable way to skip it on PyPy too
I figure we might as well :( Precedent and all that.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@coveralls

Coverage Status

Coverage increased (+0.23%) when pulling d35e5d9 on alex:switch-to-cryptography into 1af8db2 on paramiko:master.

@techtonik

Should there be an issue for PyPy tests resurrection?

@bitprophet
Member

@techtonik you mean just for the "this specific teardown test breaks on anything that isn't CPython 2.x"? Maybe. Having an issue open won't necessarily mean anybody has time to look at it :D OTOH having it clearly marked in the tracker (vs the code) as a known thing is probably good...just made #411.

@bitprophet
Member

Looks like cryptography 0.6.x is out at this point so the next spot I have time to do Paramiko things, I'll try merging this and see where it takes us

@lndbrg lndbrg commented on an outdated diff Oct 17, 2014
paramiko/ecdsakey.py
@@ -131,9 +131,7 @@ def generate(curve=curves.NIST256p, progress_func=None):
Generate a new private RSA key. This factory function can be used to
generate a new host key or authentication key.
- :param function progress_func:
- an optional function to call at key points in key generation (used
- by ``pyCrypto.PublicKey``).
+ :param function progress_func: Unused
@lndbrg
lndbrg Oct 17, 2014 Contributor

It might be worth breaking this api (e.g remove the parameter from the function), if we are going to bump to a major anyway.

@lndbrg lndbrg commented on the diff Oct 17, 2014
paramiko/rsakey.py
@@ -137,30 +161,24 @@ def generate(bits, progress_func=None):
generate a new host key or authentication key.
:param int bits: number of bits the generated key should be.
- :param function progress_func:
- an optional function to call at key points in key generation (used
- by ``pyCrypto.PublicKey``).
+ :param function progress_func: Unused
@lndbrg
lndbrg Oct 17, 2014 Contributor

Same.

@alex
alex Oct 17, 2014 Contributor

I'll wait until @bitprophet weighs in before doing that; happy to make the
change though.

On Fri, Oct 17, 2014 at 7:00 AM, Olle Lundberg notifications@github.com
wrote:

In paramiko/rsakey.py:

@@ -137,30 +161,24 @@ def generate(bits, progress_func=None):
generate a new host key or authentication key.

     :param int bits: number of bits the generated key should be.
  •    :param function progress_func:
    
  •        an optional function to call at key points in key generation (used
    
  •        by `pyCrypto.PublicKey`).
    
  •    :param function progress_func: Unused
    

Same.


Reply to this email directly or view it on GitHub
https://github.com/paramiko/paramiko/pull/394/files#r19018903.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

@lndbrg I'd rather make 2.0 a "major in name only" release, this way we can simply tell users "the only truly backwards incompat thing that changed is the dependencies". I expect we could stick out a 3.0 sometime in the next, I don't know, year or so with actual API changes.

@techtonik

Recent POODLE news say that it is ok to release things fast. =) As a user, I don't see much sense in incrementing major only for the sake of upping dependencies. From developer side it makes sense so that you can rollback easily to old deps. In that case it may be possible to release 3.0 at the same time. The only reason is management overhead.

@bitprophet
Member

@techtonik Usually the expectation is users see a major version bump and it's a decent warning that something changed they should check up on before blindly updating.

Certainly I'd rather have a 2.0 that makes some folks go "oh the API didn't change? huh ok", instead of a 1.16 that makes (probably a lot more) people wonder why they suddenly can't install because they lack OpenSSL headers.

@bitprophet
Member

@alex How much effort would it be to replace the ecdsa-using code with the equivalents from cryptography? We mentioned doing so earlier, but I think when this gets merged/released we should take care of everything in one go, that way I don't have to worry about more requirements changes later.

@alex
Contributor
alex commented Nov 7, 2014

I don't think it's a ton of work -- I can investigate in the next few days.

@alex
Contributor
alex commented Nov 8, 2014

@bitprophet Just pushed an initial implementation. There is one failing test right now, because I haven't implemented key dumping yet. Need to think about how I'll implement that, should be straightforward, just need to do it. Will try to push that later today.

@alex
Contributor
alex commented Dec 18, 2014

To update the status here: this is now feature complete, but there appears to be a bug in pyasn1 which causes the EC key serialization to not work.

@bitprophet
Member

@alex Meaning that if this gets merged we'd lose ECDSA functionality again? Or is there something we can do to mitigate/workaround?

@alex
Contributor
alex commented Dec 19, 2014

You'd lose the ability to save an ECDSA key to a file -- I don't think that
feature is part of the typical use case for paramiko (e.g. fabric); but it
would definitely be unfortunate to break it IMO. If I don't hear back from
the pyasn.1 maintainer within a few days, I'll come up with an alternate
workaround.

On Thu Dec 18 2014 at 4:21:19 PM Jeff Forcier notifications@github.com
wrote:

@alex https://github.com/alex Meaning that if this gets merged we'd
lose ECDSA functionality again? Or is there something we can do to
mitigate/workaround?


Reply to this email directly or view it on GitHub
#394 (comment).

@bitprophet
Member

Gotcha, that's less impactful than I was worrying, but yes, also less than ideal. Will wait to see what comes out of your talk w/ the pyasn.1 folks, thanks!

@alex
Contributor
alex commented Dec 19, 2014

No word back from them yet, I'll leave a note here as soon as they say
something.

On Fri Dec 19 2014 at 10:23:45 AM Jeff Forcier notifications@github.com
wrote:

Gotcha, that's less impactful than I was worrying, but yes, also less than
ideal. Will wait to see what comes out of your talk w/ the pyasn.1 folks,
thanks!


Reply to this email directly or view it on GitHub
#394 (comment).

@Julian
Julian commented Feb 9, 2015

Any chance there's been an update here?

@drewfisher314

I would love to see an update here as well. Last I heard was the proposed patch to switch to cryptography.io hit a snag in pyasn1, but I might be confused.

@alex
Contributor
alex commented Mar 3, 2015

That is correct -- the next release of cryptography will obviate that
problem though. It's due out in less than a week, so I'll update this then.

On Tue, Mar 3, 2015 at 7:14 AM, drewfisher314 notifications@github.com
wrote:

I would love to see an update here as well. Last I heard was the proposed
patch to switch to cryptography.io hit a snag in pyasn1, but I might be
confused.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@frewsxcv

Worth mentioning that a new version of Cryptography was released a few days ago: https://cryptography.io/en/latest/changelog/#id1

@alex
Contributor
alex commented Mar 11, 2015

Yup, I believe it has everything that's needed for this. I'll make time for
it before the end of the week.

On Wed, Mar 11, 2015 at 2:50 PM, Corey Farwell notifications@github.com
wrote:

Worth mentioning that a new version of Cryptography was released a few
days ago: https://cryptography.io/en/latest/changelog/#id1


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@alex
Contributor
alex commented Aug 1, 2015

@bitprophet I updated this PR with the python 3.2 stuff discussed several moons ago.

@drewfisher314

What's the current status of this PR? It seems to have gotten forgotten :(

@bitprophet
Member

@drewfisher314 it's called a day job :( this is still totally in the top few items on my list, it's just big enough (alongside the other top items like uggggh key handling) that I need to dedicate sprint time to it. Planning to spend early September in "paramiko mode".

@drewfisher314

I understand completely (I also have one of those pesky day job things). Thanks for the update!

@alex
Contributor
alex commented Nov 4, 2015

Resolved a couple of merge conflicts that had snuck in.

@techtonik

All checks have failed.

@bitprophet
Member

At least one of those is probably due to me nuking 3.2 stuff from Travis because I got fed up with it breaking things. (As a corollary I am definitely now cool with the overall removal of 3.2 support in this PR and thus Paramiko 2.0...just for the record, since I was on the fence earlier.) The others are likely to be due to the recently merged SHA-2 support.

Speaking of which, those merges mean Paramiko 1.16 is almost done, then 2.0 is the next thing up, finally.

@alex
Contributor
alex commented Nov 4, 2015

@bitprophet I'm unable to reproduce any of these failures locally, and they seem to be py3k specific. Ever seen anything like this?

@techtonik

@alex turn more verbose log / test trace on Travis? Might be a limitation of Travis Linux container.

@bitprophet
Member

Sorry I was referring to the merge conflicts, not the test failures - didn't see those at first.

Peeked at Travis and both of those issues (10-minute timeout & test_L_handshake_timeout failing) are ones I see on Travis frustratingly often for our main branches (tho most often under Python 3.2 which was on reason I nixed it from the old branches myself).

I'm also unable to get either of those to happen for me on my development workstation.

Clearly it's getting to the point where I need to take extra time to reproduce (hopefully a Linux VM + looping over the tests a bunch will suffice) though solving maybe another matter.

That said, these issues are not specific to this branch so I am not worried about the efficacy of the PR's changes, especially if the tests pass for me when I get around to merging.

Speaking of, Paramiko 1.16 is now out so as stated the next chunk of work for me on Paramiko is 2.0. I'll be hacking on that and finishing the alpha of Fabric 2.0 around the same time.

@alex
Contributor
alex commented Nov 6, 2015

Awesome, let me know if there's ever anything you need from me on finishing
this up.

On Thu, Nov 5, 2015 at 7:24 PM, Jeff Forcier notifications@github.com
wrote:

Sorry I was referring to the merge conflicts, not the test failures -
didn't see those at first.

Peeked at Travis and both of those issues (10-minute timeout &
test_L_handshake_timeout failing) are ones I see on Travis frustratingly
often for our main branches (tho most often under Python 3.2 which was on
reason I nixed it from the old branches myself).

I'm also unable to get either of those to happen for me on my development
workstation.

Clearly it's getting to the point where I need to take extra time to
reproduce (hopefully a Linux VM + looping over the tests a bunch will
suffice) though solving maybe another matter.

That said, these issues are not specific to this branch so I am not
worried about the efficacy of the PR's changes, especially if the tests
pass for me when I get around to merging.

Speaking of, Paramiko 1.16 is now out so as stated the next chunk of work
for me on Paramiko is 2.0. I'll be hacking on that and finishing the alpha
of Fabric 2.0 around the same time.


Reply to this email directly or view it on GitHub
#394 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@bitprophet
Member

Absolutely, thanks as usual @alex and apologies this has taken so long >_<

@alex
Contributor
alex commented Dec 4, 2015

merged master in because of merge conflicts. the test failure is due to #612

@techtonik

So, ping Travis again?

@bitprophet
Member

@techtonik #612 is still unresolved but its tl;dr is "Travis triggers race conditions often under Python 3" so just tickling it isn't usually enough to get a good sense of whether a PR works. Will finalize #612 before this gets merged.

@johnthagen

Just curious on the status of this PR. As someone who supports Windows, very excited to have cryptography's official Windows wheels on PyPI. Appreciate the work put into this.

@alex
Contributor
alex commented Feb 28, 2016

Resolved a merge conflict that had crept in.

@alex
Contributor
alex commented Feb 28, 2016

(test failure unrelated)

@techtonik

@alex restart test?

@johnthagen

@alex @bitprophet Just FYI, not sure if this affects paramiko directly, but PyCrypto has a known exploitable buffer overflow dlitz/pycrypto#173. Could increase the priority of porting to cryptography.

@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

@johnthagen Eek :( good to know. I was planning to expedite this anyways for other reasons (not least "I feel like a tremendous ass for letting it moulder so long"), but that's another excellent one.


So I finally got my ass in gear on #612 and think I have it licked so it'll stop making this PR (among others) harder to test.

Made my own integration branch for this ticket, and after a pip install cryptography>=0.8 I note the following things:

  • Cryptography's added a handful more sub-dependencies than I remember: on top of pyasn1 there's now pycparser, cffi, and idna.
    • Presuming those are easy to install & relatively stable, and given the payoff of merging this, I don't think I'm going to complain much. Just want it on the record. (And in the install docs? Hrm.)
  • I'm getting 3 failures/errors all from the same basic code path re: loading an RSA key (posted below).
    • Given that my setup (OS X Yosemite) isn't uncommon, this does make me worry a bit, but I'm confident it's solvable and probably something dumb like cruft in my environment somewhere.
    • If someone else sees this before I get around to debugging it and has suggestions/a fix, I am all ears. I plan to tackle this once I dig through the rest of https://github.com/paramiko/paramiko/milestones/1.16.1
  • Travis is not seeing those errors - it's happy: https://travis-ci.org/paramiko/paramiko/builds/125184508

======================================================================
ERROR: test_client_can_be_used_as_context_manager (test_client.SSHClientTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_client.py", line 318, in test_client_can_be_used_as_context_manager
    self.tc.connect(self.addr, self.port, username='slowdive', password='pygmalion')
  File "/Users/jforcier/Code/oss/paramiko/paramiko/client.py", line 367, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/Users/jforcier/Code/oss/paramiko/paramiko/client.py", line 558, in _auth
    key = pkey_class.from_private_key_file(filename, password)
  File "/Users/jforcier/Code/oss/paramiko/paramiko/pkey.py", line 196, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/Users/jforcier/Code/oss/paramiko/paramiko/rsakey.py", line 47, in __init__
    self._from_private_key_file(filename, password)
  File "/Users/jforcier/Code/oss/paramiko/paramiko/rsakey.py", line 166, in _from_private_key_file
    self._decode_key(data)
  File "/Users/jforcier/Code/oss/paramiko/paramiko/rsakey.py", line 174, in _decode_key
    data, password=None, backend=default_backend()
  File "/Users/jforcier/.virtualenvs/paramiko3/lib/python3.5/site-packages/cryptography/hazmat/primitives/serialization.py", line 28, in load_der_private_key
    return backend.load_der_private_key(data, password)
  File "/Users/jforcier/.virtualenvs/paramiko3/lib/python3.5/site-packages/cryptography/hazmat/backends/multibackend.py", line 307, in load_der_private_key
    return b.load_der_private_key(data, password)
  File "/Users/jforcier/.virtualenvs/paramiko3/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1122, in load_der_private_key
    password,
  File "/Users/jforcier/.virtualenvs/paramiko3/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1265, in _load_key
    self._handle_key_loading_error()
  File "/Users/jforcier/.virtualenvs/paramiko3/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1337, in _handle_key_loading_error
    raise ValueError("Could not unserialize key data.")
ValueError: Could not unserialize key data.
@alex
Contributor
alex commented Apr 23, 2016
  • The idna dep is newish, cffi and pycparser we've had the whole time.
  • Not sure what to make of this traceback; are you able to reproduce this issue easily locally?
@coveralls

Coverage Status

Coverage decreased (-0.3%) to 72.277% when pulling eac11dc on alex:switch-to-cryptography into a14d266 on paramiko:master.

@alex
Contributor
alex commented Apr 23, 2016

@bitprophet Ok, so that failure is probably related to your ~/.ssh/id_rsa (which I do not have locally, which is probably why I can't reproduce).

Not sure how to proceed in debugging without seeing the contents of your id_rsa, which for SUPER obvious reasons you wouldn't want to give me.

@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

Yea, that would deffo fall under "my local environment" huh. I peeked and the fact that it's being loaded at all is just due to default parameter val for Client.connect(look_for_keys). Why this wasn't causing test failures before the switch is a good question, I didn't think my RSA key was that off-spec (been using it for ages, including against relatively modern distros). It is of course passphrase protected, so if behavior around "what to do when a passphrase is found" changed, that might also do it? I welcome ideas there and will poke otherwise.

Also gonna explicitly set look_for_keys=False on these tests because this is poor behavior for a test suite IMHO.

@alex
Contributor
alex commented Apr 23, 2016

Unfortunately I can't think of a good way for me to debug without you
sharing the key. Can you try generating a fresh key, putting it there, and
if that triggers the error sharing that key?

On Sat, Apr 23, 2016 at 12:39 PM, Jeff Forcier notifications@github.com
wrote:

Yea, that would deffo fall under "my local environment" huh. I peeked and
the fact that it's being loaded at all is just due to default parameter val
for Client.connect(look_for_keys). Why this wasn't causing test failures
before the switch is a good question, I didn't think my RSA key was that
off-spec (been using it for ages, including against relatively modern
distros). I welcome ideas there and will poke otherwise.

Also gonna explicitly set look_for_keys=False on these tests because this
is poor behavior for a test suite IMHO.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#394 (comment)

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

@bitprophet
Member

Yea, I'll do that, and also going to try comparing what I can between it & some of the keys stored in the test suite.

Amusingly I was going to regenerate new keys for myself soon anyways, but I figure if a key I presumably generated in a vanilla fashion a while back is "broken", other users might encounter this too, so still want to figure out WTF.

@bitprophet bitprophet added a commit that referenced this pull request Apr 23, 2016
@bitprophet bitprophet Set look_for_keys=False in client tests to avoid loading real user keys.
Re #394 but also feels like good practice anyways
403dac2
@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

Digging more, suspect the reason this fails on this branch and not master isn't the key, it's the type of exception raised; most of the internals of Paramiko have try / except SSHException sprinkled around, and failure to load a passphrase-protected key raises SSHException subclasses; so such exceptions get caught, stored, and then only if all attempts to auth fail, are any of those bubbled up. (This is very much re: #387).

In the test suite, it's almost all password auth, so what happens is keys are tried & fail & are muted, then password auth is tried, succeeds, and the earlier errors are discarded.

In this PR's code, due to the new backend, the attempt to load the personal/protected key is raising ValueError, which Paramiko isn't tuned to expect, so it bubbles up immediately, not allowing the "trickle down" to password auth to occur.

I'll look deeper now to see what the best solution probably is. I loathe SSHException but was hoping to save tweaking it for later, really don't want it to be a blocker on this. (Honestly, ditto #387 - I'd like to release this PR sooner instead of later, even if that means making it a non-2.0 release, or making it 2.0 and then #387 and friends being a 3.0.)

Hoping we can extend a low-level except to also skip/store ValueError & pray that doesn't make it cast too wide a net.

@alex
Contributor
alex commented Apr 23, 2016

Ahhhh, and I bet there's no tests for "failure to parse raises an SSHError", it sounds like the key loading should just be written in a try / except ValueError: raise SSHError?

@bitprophet
Member

Yea pretty much. We'll see!

@bitprophet
Member

Also, you're so fast on responding to updates that I need to curb my "ninja edit" habit. :D :D plz do see the parenthetical above about release stuff.

@bitprophet
Member

Also, shoutout to how the ValueError-raising statement is on line 1337 of the openssl backend. 👍

@coveralls

Coverage Status

Coverage decreased (-0.3%) to 72.298% when pulling f133db6 on alex:switch-to-cryptography into a14d266 on paramiko:master.

@coveralls

Coverage Status

Coverage decreased (-0.3%) to 72.255% when pulling 57c1635 on alex:switch-to-cryptography into a14d266 on paramiko:master.

@alex
Contributor
alex commented Apr 23, 2016

@bitprophet should be fully resolved now!

@bitprophet
Member

Looks ok, testing locally :) We ideally want a more useful message in the exception than "Unable to deserialize" (something saying "I tried deserializing this key with this password and it failed") but I think that can be considered part of #387.

Anyway...

  • Nabbed latest, confirm test suite still happy (because of my earlier changes)
  • Confirmed my temp reproduction code triggering the issue is now happy (insofar as it raises the same SSHException based "couldn't make any auth work" exception chain as master)
    • Temp-reverted my earlier fixes to the test suite re: look_for_keys, confirmed test suite still happy.
  • Created a new, real test because manual testing is dumb & it makes the test suite that much closer to a description of expected behavior
    • Annoyingly, this ValueError business doesn't happen with the test suite's password-protected DSA key, but does happen with the RSA key. Was not expecting this, started out using the DSA key arbitrarily, and that wasted a lot of my time 😢
    • Tested against this PR branch prior to alex's recent fixes; confirmed fail
    • Tested against latest HEAD of this branch; confirmed pass
    • Tested against master; confirmed pass
@bitprophet bitprophet added a commit that referenced this pull request Apr 23, 2016
@bitprophet bitprophet Add regression test protecting against an issue found in #394.
Putting it in prior to merge of #394 because it also serves as
a good explicit test of behavior which was previously implicit
8057fca
@alex
Contributor
alex commented Apr 23, 2016

@bitprophet Awesome. What's the next step here?

@Julian
Julian commented Apr 23, 2016

Champagne. Is it champagne time? I'll buy?
On Apr 23, 2016 15:13, "Alex Gaynor" notifications@github.com wrote:

@bitprophet https://github.com/bitprophet Awesome. What's the next step
here?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#394 (comment)

@coveralls

Coverage Status

Coverage decreased (-0.4%) to 72.247% when pulling d1e43c1 on alex:switch-to-cryptography into 8057fca on paramiko:master.

@bitprophet
Member

@alex @Julian I deffo want to get this out ASAP, what remains as far as I can tell:

  • I gotta make sure I personally understand the installation & upgrade ramifications, both generally (I had the mistaken assumption this was all pure-Python and not C extensions. I blame me being stupid.) and re: the handful of newer dependencies.
    • I expect this to just be me testing installation a few times in a few envs, & maybe tweaking the install docs a bit.
  • Need to decide whether this lives in 1.17 or 2.0 as previously planned. Still leaning 2.0, but in a sense where the rest of the current 2.0 milestone gets bumped to 3.0 (with 3.0 being slated for "still pretty soon").
    • tl;dr I want this to get out in the wild without having to wait for the technically-unrelated other stuff like #387, which could take more time to hammer out
    • but there's still that ingrained reluctance to put out "too many" major revisions "too soon". I think this is mostly bunk though, esp since a 2.0 with just this PR merge is fully API compatible with 1.0.
      • But using major versioning to signal "a big potentially disruptive change" still seems like a better idea than "oh yea this is just another 1.N feature release".
  • Want to get 1.16.1 out first to relieve bugfix pressure on the 1.x line. Shouldn't take more than a day or two (& if it does I will just punt stuff).
    • Will probably merge #394 to master before then, just to make it easier for bleeding edge folks to try it out.
@reaperhulk

@bitprophet Yeah there are definitely some nuances since cryptography links against OpenSSL (and CommonCrypto on OS X). A few things to keep in mind:

Cryptography is currently planning to drop OpenSSL 0.9.8 support in its next release as well. This should only affect extremely old distributions such as RHEL/CentOS 5, but we don't want to catch anyone by surprise. For the 1.4 release (probably released next month) there will be a workaround, but if you feel strongly about continuing 0.9.8 support let us know :)

@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

Notes to myself...

Install testing

  • Installing cryptography in a vanilla Python 2.7 (OS X Yosemite bundled version; I have the usual Python build headers & openssl headers installed too) virtualenv using pip 7.1.2 results in the following:
» pip freeze
cffi==1.6.0
cryptography==1.3.1
enum34==1.1.3
idna==2.1
ipaddress==1.0.16
pyasn1==0.1.9
pycparser==2.14
six==1.10.0
wheel==0.24.0
  • cryptography and cffi download as tgz and then run bdist_wheel; the rest download as whl directly.
  • After blowing away my virtualenv, recreating it, and upgrade pip to latest 8.x (EDIT: I noticed this myself from crypto's docs, but @reaperhulk confirms above too, thx!), confirm that both cryptography and cffi download & install wheels directly instead.
  • On a Debian 8 vm which previously had python-dev installed, also with pip 7.1.2, as expected had to install libffi-dev and libssl-dev for those two wheels to build successfully.
    • Further, confirmed that a workable up-to-date dev virtualenv for paramiko master passes tests, then checked out this branch, pip install -e . to receive the upgraded dependencies, then confirmed tests continue to pass.
      • And nuking pycrypto from the virtualenv has no observable result, as expected.

Stuff to tweak in paramiko

(I can handle these...)

  • Because the install docs are part of the www and not docs site, I'll need to shuffle things around so the 1.x version of the doc is still available in some form.
    • Could be an argument for moving the install doc to the (versioned) docs site...
    • Otherwise, simply moving 1.x specific stuff to another page or section, should suffice. I don't expect installation to change much after this, even in 3.0 or 4.0 or whatnot - those will be API or behavioral changes.
  • Previously, pyasn1 is an optional dependency, for the GSSAPI stuff. That language needs to be removed now that pyasn1 is a hard requirement as a sub-dependency of crypto.io.
    • This branch lists pyasn1 as a hard dep, but I think it may be easier for all concerned to simply defer to the crypto.io docs (with a warning that it includes "a handful of C extensions and pure-python modules" or w/e) instead of trying to mirror the full list of sub-dependencies, having them get out of date, etc.
    • Another good argument for deferring is that the language in this PR's version of the install doc, about needing to build from source, lacks the updated info (again per crypto's docs and @reaperhulk) about pip 8 enabling static wheels on OS X. So if we don't defer, this needs an update too.
  • Consider adding an "upgrading from 1 to 2" section to the install docs; this would also make it a good place to note that ecdsa is no longer required (otherwise that should at least go in the changelog?)
  • Not 100% sure how to handle the PyPM section given PyPM may not upgrade their side of things immediately (?) but leaning "leave it as-is, direct any complaints to PyPM if it comes up"
@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

Poking the docs now...almost forgot one thing though. @reaperhulk / @alex I'm wondering if there's a way to avoid the potential 'shock' of the 0.9.8 version drop somehow (but no I don't think its support should be kept, offhand).

  • If I get Paramiko 2.0 out in the next few days, and leave its setup data saying cryptography>=0.8, this means users would be capable of installing it + crypto 1.3 + openssl 0.9.8, then getting screwed the next time they reinstall their env or pip install -U after crypto 1.4 comes out.
  • I could switch to cryptography >=0.8,<1.4, which would prevent that scenario initially.
    • I imagine following this up by changing that restriction, with a warning in the changelog, in e.g. 2.1 or 2.2. Lets us move away from openssl 0.9.8 but with somewhat less surprise.
  • Alternately: stick with >=0.8, but put a warning in my own docs mirroring Crypto's, noting "heyyy openssl 0.9.8 sucks don't use it or you'll be sorryyyy".
    • Asses covered, less changes required down the road. Still potentially surprising for folks who don't read docs. Meh?
  • Alternately again: wait until Crypto 1.4 is out before releasing Paramiko 2.0.
    • This might also let me have time to slap more backwards incompat tickets into Paramiko master
    • Downside: means folks waiting for this poor ticket are waiting even longer; slight danger of me being pulled into other things in the interim.
  • Worst-case scenario: like half my users hear about these plans re: OpenSSL 0.9.8, throw a pity party about it because they're still on CentOS 5 or whatever, and I ponder leaning into your workaround or requesting longer-term 0.9.8 support.
    • I don't see this being that likely; I do have a surprisingly large number of folks stuck on Python 2.6, but that by definition excludes users on stock CentOS 5 (which is Python 2.4). If someone is on CentOS 5 and backported Python 2.6, surely they can backport OpenSSL.

I'm leaning towards the third option, stick with crypto >=0.8 only, put a warning block in my own docs about OpenSSL 0.9.8, say good enough. Thoughts?

@alex
Contributor
alex commented Apr 23, 2016

FWIW, by the numbers (NB: we only have OpenSSL version data for folks on a very recent pip): In the past 2 weeks, 0.3% of installs of paramiko used OpenSSL 0.9.8 (161,004 download sample).

@reaperhulk

I'm in favor of option 3 myself. 0.9.8 is over 8 years old and its mere existence is holding back the security of the internet 😠

@bitprophet
Member

Works for me, thanks both!

Mostly done with the docs treatment now, will push my integration branch when that's complete, though that's more for reference, I don't care hard whether you merge it, I can just as easily merge it + this into master when necessary.

@bitprophet bitprophet added a commit that referenced this pull request Apr 23, 2016
@bitprophet bitprophet Tweaks to install docs re #394 41f6f4b
@bitprophet bitprophet added a commit that referenced this pull request Apr 23, 2016
@bitprophet bitprophet Tweaks to install docs re #394 37294ee
@bitprophet
Member
bitprophet commented Apr 23, 2016 edited

Neglected to tag this on another commit re: docs...after some rebasing and force pushing to my integration branch, here's a SHA-pinned comparison, cc @alex @reaperhulk: alex@alex:d1e43c1...paramiko:0ae4d44#diff-d3b5c8a3c496304a374ce4a27034e686

@reaperhulk

Mostly looks good. Only comment I have is that "plus development headers for Python, OpenSSL and CFFI" should probably read "plus development headers for Python, OpenSSL and libffi", as libffi is cffi's C dependency.

@bitprophet
Member

Updated, thanks!

@bitprophet
Member

Posted http://bitprophet.org/blog/2016/04/23/paramiko-2.0-is-coming/ so this doesn't catch everyone 100% flat-footed when it comes out :3

I'm off to bang on 1.16.1; some of its tickets may get bumped to 1.17.0; once those are both squared away I am probably going to declare master "2.0" and merge this.

@coveralls

Coverage Status

Coverage decreased (-0.3%) to 72.326% when pulling 69b995a on alex:switch-to-cryptography into 1cda0eb on paramiko:master.

@coveralls

Coverage Status

Coverage decreased (-0.3%) to 72.339% when pulling 69b995a on alex:switch-to-cryptography into 1cda0eb on paramiko:master.

@bitprophet bitprophet merged commit 69b995a into paramiko:master Apr 25, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@alex alex deleted the alex:switch-to-cryptography branch Apr 25, 2016
@alex
Contributor
alex commented Apr 25, 2016

\o/

Thanks! Feel free to CC me on any issues should there be follow up necessary.

@bitprophet
Member

Did initial prep work for release:

  • Branched old master as 1.17 so that can be released (it's tiny, 1.16.1 will be much bigger changelog-wise)
  • Updated my integration branch (which has some of its own changes, as above, mostly docs) with alex's branch and master (been working on bugfixes a lot today). Looks ok
  • Merged that to master and pushed, which is why this is merged now :3

I plan to release 1.16.1, 1.17.0 and 2.0.0 tomorrow if all goes well. Thanks a billion, Alex, you have the patience of a saint.

@bitprophet
Member

Feel free to CC me on any issues should there be follow up necessary.

You know I will 😆

@drewfisher314

Outstanding!

On Apr 24, 2016, at 9:21 PM, Jeff Forcier notifications@github.com wrote:

Merged #394.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@bitprophet bitprophet added a commit that referenced this pull request Apr 25, 2016
@bitprophet bitprophet Changelog re #394 c2d0aa8
@techtonik

👍

@Julian
Julian commented Apr 25, 2016

@alex @bitprophet and others, all y'all are great. Thanks!

@bitprophet
Member

Quick note, still planning to get this out ASAP, been banging on an update to my changelog library so it can actually support 1.0 -> 2.0 transitions gracefully. Mostly done with that now. Computers.

@alex
Contributor
alex commented Apr 27, 2016

@bitprophet Sounds good, let me know if you need anything.

@florianluediger florianluediger added a commit to instantshare/instantshare that referenced this pull request May 16, 2016
@florianluediger florianluediger Update installation instructions for Windows.
Paramiko has switched from PyCrypto to Cryptography as backend.
This makes the installation in Windows much easier.
See paramiko/paramiko#394 for more information.
bd492c2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment