Large performance regression at SSHClient.load_system_host_keys() since paramiko 1.12.0

[jan-matejka]

Large performance regression at SSHClient.connect() since paramiko 1.12.0

Details (including reprodcer) at https://bugzilla.novell.com/show_bug.cgi?id=891383

[jan-matejka]

Turns out the violatior is load_system_host_keys(), not connect()

new sscce:

import paramiko
import datetime

now = datetime.datetime.now

print "paramiko: {0}".format(paramiko.__version__)

s = now()

client = paramiko.SSHClient()
print "SSHClient(): {0}".format(now()-s)
s = now()

client.load_system_host_keys()
print "load_system_host_keys(): {0}".format(now()-s)
s = now()

client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
print "set_missing_host_key_policy(): {0}".format(now()-s)
s = now()

client.connect("10.121.224.178", 22, username='root')
print "connect(): {0}".format(now()-s)

The numbers:

on opensuse:

yac@linux-1e2q % for i in {0..4}; do python paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.11.0
SSHClient(): 0:00:00.000037
load_system_host_keys(): 0:00:00.273131
set_missing_host_key_policy(): 0:00:00.000021
connect(): 0:00:01.241252

paramiko: 1.11.0
SSHClient(): 0:00:00.000031
load_system_host_keys(): 0:00:00.270977
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.235447

paramiko: 1.11.0
SSHClient(): 0:00:00.000034
load_system_host_keys(): 0:00:00.359807
set_missing_host_key_policy(): 0:00:00.000017
connect(): 0:00:01.207291

paramiko: 1.11.0
SSHClient(): 0:00:00.000043
load_system_host_keys(): 0:00:00.418801
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.245537

paramiko: 1.11.0
SSHClient(): 0:00:00.000032
load_system_host_keys(): 0:00:00.284946
set_missing_host_key_policy(): 0:00:00.000015
connect(): 0:00:01.239951

---

~
yac@linux-1e2q % for i in {0..4}; do python paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.13.1
SSHClient(): 0:00:00.000036
load_system_host_keys(): 0:00:09.210335
set_missing_host_key_policy(): 0:00:00.000021
connect(): 0:00:01.242118
python paramiko-sscce.py 6.79s user 0.04s system 61% cpu 11.047 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000039
load_system_host_keys(): 0:00:08.449971
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.236512
python paramiko-sscce.py 6.82s user 0.02s system 66% cpu 10.218 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000036
load_system_host_keys(): 0:00:09.209746
set_missing_host_key_policy(): 0:00:00.000014
connect(): 0:00:01.244098
python paramiko-sscce.py 6.84s user 0.02s system 64% cpu 10.728 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000038
load_system_host_keys(): 0:00:09.368015
set_missing_host_key_policy(): 0:00:00.000017
connect(): 0:00:01.239716
python paramiko-sscce.py 6.82s user 0.06s system 63% cpu 10.822 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000031
load_system_host_keys(): 0:00:07.497197
set_missing_host_key_policy(): 0:00:00.000017
connect(): 0:00:01.251877
python paramiko-sscce.py 6.86s user 0.02s system 76% cpu 8.949 total

on gentoo:

yac@deathstar % for i in {0..4}; do python paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.11.0
SSHClient(): 0:00:00.000016
load_system_host_keys(): 0:00:00.152889
set_missing_host_key_policy(): 0:00:00.000006
connect(): 0:00:00.793254

paramiko: 1.11.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:00.151677
set_missing_host_key_policy(): 0:00:00.000006
connect(): 0:00:00.759362

paramiko: 1.11.0
SSHClient(): 0:00:00.000016
load_system_host_keys(): 0:00:00.150912
set_missing_host_key_policy(): 0:00:00.000006
connect(): 0:00:00.805230

paramiko: 1.11.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:00.152510
set_missing_host_key_policy(): 0:00:00.000006
connect(): 0:00:00.742327

paramiko: 1.11.0
SSHClient(): 0:00:00.000019
load_system_host_keys(): 0:00:00.150648
set_missing_host_key_policy(): 0:00:00.000009
connect(): 0:00:00.745299

---

~
yac@deathstar % for i in {0..4}; do python paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.13.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:03.074292
set_missing_host_key_policy(): 0:00:00.000007
connect(): 0:00:00.748837

paramiko: 1.13.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:03.116928
set_missing_host_key_policy(): 0:00:00.000008
connect(): 0:00:00.739140

paramiko: 1.13.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:03.170948
set_missing_host_key_policy(): 0:00:00.000007
connect(): 0:00:00.764470

paramiko: 1.13.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:03.123453
set_missing_host_key_policy(): 0:00:00.000008
connect(): 0:00:00.793818

paramiko: 1.13.0
SSHClient(): 0:00:00.000015
load_system_host_keys(): 0:00:03.121765
set_missing_host_key_policy(): 0:00:00.000008
connect(): 0:00:00.767693

[jan-matejka]

yac@deathstar % cat .ssh/known_hosts| wc -l
201
yac@linux-1e2q % cat .ssh/known_hosts| wc -l
214

both hashed.

[lndbrg]

This is most likely due to hashing of the hostkeys. There is a patch here #272 that you can try.

[jan-matejka]

I applied 1795d9b but it doesn't seem to help much:

yac@linux-1e2q % for i in {0..4}; do python paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.13.1
SSHClient(): 0:00:00.000041
load_system_host_keys(): 0:00:07.877842
set_missing_host_key_policy(): 0:00:00.000015
connect(): 0:00:01.249108
python paramiko-sscce.py 6.81s user 0.02s system 72% cpu 9.473 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000055
load_system_host_keys(): 0:00:09.245993
set_missing_host_key_policy(): 0:00:00.000022
connect(): 0:00:01.236385
python paramiko-sscce.py 6.78s user 0.04s system 63% cpu 10.698 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000046
load_system_host_keys(): 0:00:09.148316
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.238525
python paramiko-sscce.py 6.80s user 0.02s system 64% cpu 10.605 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000035
load_system_host_keys(): 0:00:08.291643
set_missing_host_key_policy(): 0:00:00.000018
connect(): 0:00:01.239652
python paramiko-sscce.py 6.79s user 0.02s system 69% cpu 9.746 total

paramiko: 1.13.1
SSHClient(): 0:00:00.000036
load_system_host_keys(): 0:00:08.015618
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.243820
python paramiko-sscce.py 6.75s user 0.06s system 71% cpu 9.476 total

[jan-matejka]

this is what I get when installing paramiko git 991d56b into virtualenv

yac@linux-1e2q % for i in {0..4}; do python ~/paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.14.0
SSHClient(): 0:00:00.000025
load_system_host_keys(): 0:00:08.779746
set_missing_host_key_policy(): 0:00:00.000023
connect(): 0:00:01.249543
python ~/paramiko-sscce.py 6.82s user 0.04s system 65% cpu 10.416 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000046
load_system_host_keys(): 0:00:08.291432
set_missing_host_key_policy(): 0:00:00.000020
connect(): 0:00:01.239235
python ~/paramiko-sscce.py 6.82s user 0.03s system 68% cpu 10.055 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000045
load_system_host_keys(): 0:00:10.093312
set_missing_host_key_policy(): 0:00:00.000019
connect(): 0:00:01.259932
python ~/paramiko-sscce.py 6.82s user 0.03s system 57% cpu 11.888 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000029
load_system_host_keys(): 0:00:09.253017
set_missing_host_key_policy(): 0:00:00.000020
connect(): 0:00:01.258569
python ~/paramiko-sscce.py 6.84s user 0.03s system 62% cpu 10.956 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000029
load_system_host_keys(): 0:00:08.631883
set_missing_host_key_policy(): 0:00:00.000015
connect(): 0:00:01.248560
python ~/paramiko-sscce.py 6.85s user 0.03s system 66% cpu 10.353 total

and the same with 1795d9b applied:

yac@linux-1e2q % for i in {0..4}; do python ~/paramiko-sscce.py; sleep 1; echo ""; done
paramiko: 1.14.0
SSHClient(): 0:00:00.000028
load_system_host_keys(): 0:00:08.263340
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.239470
python ~/paramiko-sscce.py 6.74s user 0.02s system 68% cpu 9.937 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000028
load_system_host_keys(): 0:00:08.222772
set_missing_host_key_policy(): 0:00:00.000017
connect(): 0:00:01.239331
python ~/paramiko-sscce.py 6.77s user 0.02s system 68% cpu 9.871 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000031
load_system_host_keys(): 0:00:09.697254
set_missing_host_key_policy(): 0:00:00.000030
connect(): 0:00:01.238208
python ~/paramiko-sscce.py 6.75s user 0.04s system 59% cpu 11.379 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000048
load_system_host_keys(): 0:00:09.025956
set_missing_host_key_policy(): 0:00:00.000019
connect(): 0:00:01.243423
python ~/paramiko-sscce.py 6.69s user 0.03s system 63% cpu 10.618 total

paramiko: 1.14.0
SSHClient(): 0:00:00.000027
load_system_host_keys(): 0:00:07.451757
set_missing_host_key_policy(): 0:00:00.000015
connect(): 0:00:01.234648
python ~/paramiko-sscce.py 6.70s user 0.02s system 73% cpu 9.164 total

[lndbrg]

Would you mind using git bisect to find the problematic changeset?

Could you also provide a profile of your program?
https://docs.python.org/2/library/profile.html#module-cProfile

Thanks!

[jan-matejka]

yac@linux-1e2q % python paramiko-profile.py
paramiko: 1.13.1
         4829514 function calls in 11.162 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   11.162   11.162 <string>:1(<module>)
     7307    0.011    0.000    0.024    0.000 HMAC.py:131(update)
     7307    0.032    0.000    0.087    0.000 HMAC.py:166(digest)
     7307    0.111    0.000    0.460    0.000 HMAC.py:82(__init__)
    29228    0.062    0.000    0.187    0.000 SHA.py:73(__init__)
    14614    0.019    0.000    0.091    0.000 SHA.py:76(new)
    14614    0.063    0.000    0.270    0.000 SHA.py:79(new)
        1    0.000    0.000    0.000    0.000 UserDict.py:18(__getitem__)
        1    0.000    0.000    0.000    0.000 UserDict.py:70(__contains__)
      214    0.001    0.000    0.007    0.000 __init__.py:1008(getLogger)
        1    0.000    0.000    0.000    0.000 __init__.py:1054(_fixupParents)
        1    0.000    0.000    0.000    0.000 __init__.py:1110(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:1140(info)
        1    0.000    0.000    0.000    0.000 __init__.py:1329(getEffectiveLevel)
        1    0.000    0.000    0.000    0.000 __init__.py:1343(isEnabledFor)
      214    0.001    0.000    0.007    0.000 __init__.py:1550(getLogger)
        1    0.000    0.000    0.000    0.000 __init__.py:182(_checkLevel)
      214    0.001    0.000    0.003    0.000 __init__.py:210(_acquireLock)
      214    0.000    0.000    0.002    0.000 __init__.py:219(_releaseLock)
        1    0.000    0.000    0.000    0.000 __init__.py:49(normalize_encoding)
        1    0.000    0.000    0.000    0.000 __init__.py:578(__init__)
      214    0.000    0.000    0.000    0.000 __init__.py:584(addFilter)
        1    0.000    0.000    0.000    0.000 __init__.py:71(search_function)
        1    0.000    0.000    0.000    0.000 __init__.py:952(__init__)
      951    0.004    0.000    0.004    0.000 _weakrefset.py:36(__init__)
      317    0.012    0.000    0.065    0.000 abc.py:86(__new__)
      317    0.002    0.000    0.006    0.000 abc.py:89(<genexpr>)
    14614    0.069    0.000    0.148    0.000 base64.py:310(encodestring)
     7520    0.026    0.000    0.039    0.000 base64.py:319(decodestring)
        1    0.000    0.000    0.000    0.000 client.py:60(__init__)
        1    0.000    0.000   11.162   11.162 client.py:72(load_system_host_keys)
        1    0.000    0.000    0.000    0.000 codecs.py:77(__new__)
       69    0.001    0.000    9.151    0.133 ecdsa.py:190(point_is_valid)
       69    0.001    0.000    0.001    0.000 ecdsa.py:76(__init__)
       69    0.001    0.000    9.162    0.133 ecdsakey.py:41(__init__)
      207    0.128    0.001    9.148    0.044 ellipticcurve.py:109(__mul__)
       69    0.005    0.000    0.005    0.000 ellipticcurve.py:112(leftmost_bit)
      138    0.000    0.000    9.148    0.066 ellipticcurve.py:140(__rmul__)
    17664    0.368    0.000    7.681    0.000 ellipticcurve.py:149(double)
      138    0.000    0.000    0.000    0.000 ellipticcurve.py:168(x)
      138    0.000    0.000    0.000    0.000 ellipticcurve.py:171(y)
      138    0.000    0.000    0.000    0.000 ellipticcurve.py:174(curve)
      138    0.000    0.000    0.000    0.000 ellipticcurve.py:177(order)
    20976    0.010    0.000    0.010    0.000 ellipticcurve.py:48(p)
    17664    0.008    0.000    0.008    0.000 ellipticcurve.py:51(a)
    21183    0.176    0.000    0.176    0.000 ellipticcurve.py:57(contains_point)
    21114    0.100    0.000    0.276    0.000 ellipticcurve.py:66(__init__)
    24564    0.022    0.000    0.022    0.000 ellipticcurve.py:76(__eq__)
     3312    0.055    0.000    1.333    0.000 ellipticcurve.py:85(__add__)
    29228    0.054    0.000    0.125    0.000 hashalgo.py:34(__init__)
    21921    0.018    0.000    0.048    0.000 hashalgo.py:53(update)
     7307    0.005    0.000    0.014    0.000 hashalgo.py:71(digest)
     7307    0.026    0.000    0.030    0.000 hashalgo.py:92(copy)
      317    0.110    0.000    1.933    0.006 hostkeys.py:124(lookup)
      317    0.002    0.000    0.002    0.000 hostkeys.py:133(SubDict)
      317    0.001    0.000    1.934    0.006 hostkeys.py:185(check)
     7307    0.185    0.000    1.084    0.000 hostkeys.py:255(hash_host)
      213    0.000    0.000    0.000    0.000 hostkeys.py:289(__init__)
      214    0.004    0.000    9.215    0.043 hostkeys.py:294(from_line)
        2    0.000    0.000    0.000    0.000 hostkeys.py:49(__init__)
        1    0.012    0.012   11.162   11.162 hostkeys.py:76(load)
       69    0.000    0.000    0.000    0.000 keys.py:19(__init__)
       69    0.001    0.000    0.002    0.000 keys.py:23(from_public_point)
       69    0.002    0.000    9.157    0.133 keys.py:32(from_string)
     1278    0.002    0.000    0.003    0.000 message.py:101(get_bytes)
      639    0.001    0.000    0.004    0.000 message.py:144(get_size)
      288    0.001    0.000    0.015    0.000 message.py:161(get_mpint)
      282    0.001    0.000    0.006    0.000 message.py:177(get_text)
      357    0.001    0.000    0.003    0.000 message.py:189(get_binary)
      213    0.000    0.000    0.001    0.000 message.py:43(__init__)
    20907    6.206    0.000    8.279    0.000 numbertheory.py:186(inverse_mod)
        1    0.000    0.000   11.162   11.162 paramiko-profile.py:4(sscce)
        1    0.000    0.000    0.000    0.000 posixpath.py:258(expanduser)
    14828    0.012    0.000    0.016    0.000 py3compat.py:36(b)
    14896    0.020    0.000    0.110    0.000 py3compat.py:46(u)
     7307    0.006    0.000    0.010    0.000 py3compat.py:68(bchr)
      144    0.002    0.000    0.040    0.000 rsakey.py:43(__init__)
      214    0.001    0.000    0.002    0.000 threading.py:146(acquire)
      214    0.001    0.000    0.001    0.000 threading.py:186(release)
      428    0.000    0.000    0.000    0.000 threading.py:63(_note)
    14896    0.016    0.000    0.045    0.000 utf_8.py:15(decode)
        1    0.000    0.000    0.000    0.000 utf_8.py:33(getregentry)
      144    0.004    0.000    0.019    0.000 util.py:127(bit_length)
      138    0.001    0.000    0.001    0.000 util.py:174(string_to_number)
      214    0.001    0.000    0.008    0.000 util.py:269(get_logger)
     7307    0.487    0.000    0.644    0.000 util.py:316(constant_time_bytes_eq)
      288    0.010    0.000    0.012    0.000 util.py:39(inflate_long)
      144    0.012    0.000    0.014    0.000 util.py:62(deflate_long)
    14614    0.007    0.000    0.007    0.000 {Crypto.Util.strxor.strxor_c}
        1    0.000    0.000    0.000    0.000 {__import__}
    14896    0.029    0.000    0.029    0.000 {_codecs.utf_8_decode}
    29228    0.023    0.000    0.023    0.000 {_hashlib.openssl_sha1}
     5661    0.002    0.000    0.002    0.000 {_struct.pack}
     6583    0.002    0.000    0.002    0.000 {_struct.unpack}
     7520    0.013    0.000    0.013    0.000 {binascii.a2b_base64}
    14614    0.031    0.000    0.031    0.000 {binascii.b2a_base64}
      138    0.000    0.000    0.000    0.000 {binascii.hexlify}
      318    0.039    0.000    0.039    0.000 {built-in method __new__ of type object at 0x7fd5f83f4d20}
      213    0.001    0.000    0.001    0.000 {cStringIO.StringIO}
     7307    0.005    0.000    0.005    0.000 {chr}
  3137514    2.073    0.000    2.073    0.000 {divmod}
     6023    0.007    0.000    0.007    0.000 {getattr}
    29229    0.048    0.000    0.048    0.000 {hasattr}
    30368    0.008    0.000    0.008    0.000 {isinstance}
    62195    0.026    0.000    0.026    0.000 {len}
      214    0.000    0.000    0.000    0.000 {method 'acquire' of 'thread.lock' objects}
    14828    0.011    0.000    0.011    0.000 {method 'append' of 'list' objects}
     7307    0.004    0.000    0.004    0.000 {method 'copy' of '_hashlib.HASH' objects}
    14896    0.042    0.000    0.087    0.000 {method 'decode' of 'str' objects}
    14614    0.015    0.000    0.015    0.000 {method 'digest' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
        2    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      317    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
    14615    0.012    0.000    0.012    0.000 {method 'join' of 'str' objects}
     1278    0.001    0.000    0.001    0.000 {method 'read' of 'cStringIO.StringI' objects}
      214    0.000    0.000    0.000    0.000 {method 'release' of 'thread.lock' objects}
     7307    0.015    0.000    0.015    0.000 {method 'replace' of 'unicode' objects}
        2    0.000    0.000    0.000    0.000 {method 'rfind' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'rstrip' of 'str' objects}
     7736    0.021    0.000    0.021    0.000 {method 'split' of 'str' objects}
    65093    0.031    0.000    0.031    0.000 {method 'startswith' of 'str' objects}
      214    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}
    29228    0.035    0.000    0.035    0.000 {method 'update' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {open}
   877272    0.144    0.000    0.144    0.000 {ord}
    14902    0.019    0.000    0.019    0.000 {range}
      428    0.000    0.000    0.000    0.000 {thread.get_ident}

yac@carolyn % cat paramiko-profile.py 
import paramiko
import cProfile

def sscce():
    print "paramiko: {0}".format(paramiko.__version__)
    client = paramiko.SSHClient()
    client.load_system_host_keys()

cProfile.run("sscce()")

[jan-matejka]

yac@linux-1e2q % cat paramiko-bisect.py 
import paramiko
import datetime
import cProfile
import sys

now = datetime.datetime.now

def sscce():
    print "paramiko: {0}".format(paramiko.__version__)

    s = now()

    client = paramiko.SSHClient()
    print "SSHClient(): {0}".format(now()-s)
    s = now()

    client.load_system_host_keys()
    d = now()-s
    print "load_system_host_keys(): {0}".format(d)

    if d > datetime.timedelta(0, 4):
        sys.exit(1)

    sys.exit(0)

sscce()

yac@linux-1e2q % cat paramiko-bisect.sh 
#! /usr/bin/env sh

GIT_WD=`pwd -P`
VENV=/tmp/parmaiko-venv
if test -d ${VENV}; then
    rm -rf ${VENV}
fi
virtualenv-2.7 ${VENV}
. ${VENV}/bin/activate
python setup.py install
python ~/paramiko-bisect.py

on

g bisect start v1.13.0 v1.11.0
g bisect run ~/paramiko-bisect.sh

gives

Finished processing dependencies for paramiko==1.10.0
paramiko: 1.10.0
SSHClient(): 0:00:00.000033
load_system_host_keys(): 0:00:00.024095
ebdbfae5b1fd069f53c13b830fcf737b279977b7 is the first bad commit
commit ebdbfae5b1fd069f53c13b830fcf737b279977b7
Author: Ethan Glasser-Camp <ethan@betacantrips.com>
Date:   Mon Mar 25 11:40:46 2013 -0400

    Hook up ECDSA to hostkeys

    More sophisticated key negotiation is still necessary in the case
    where we have an ECDSA key for the server and it offers us both RSA
    and ECDSA. In this case, we will pick RSA and fail because we don't
    have it. Instead, we should pick ECDSA. Still, this works if you tell
    your server to only offer ECDSA keys :)

:040000 040000 9716f3f55a792e2bd8bb8cbf15557a2c2709be13 fddc8eefafd55b4b526fcfc1dbfe27ecf0bc14b6 M      paramiko
bisect run success

the "paramiko: 1.10.0" output is a little confusing to me but I guess it has something to do with the git merges.

[lndbrg]

Yeah, that was my second suspicion. Also given your profile it looked to be ECDSA related. Are your server offering ECDSA keys?

[jan-matejka]

ssh -vvv <host> gives

debug1: Host 'cunningham.qam.suse.de' is known and matches the ECDSA host key.

so I guess so.

[lndbrg]

Can you try #298 i think this will fix it since you spend a lot of time in point_is_valid

[jan-matejka]

1.13.0 with spearki@77b1aac applied looks a lot better:

yac@linux-1e2q % for i in {0..4}; do python ~/paramiko-sscce.py; sleep 1; echo; done                                                                                                                                                                                             
paramiko: 1.13.0
SSHClient(): 0:00:00.000031
load_system_host_keys(): 0:00:01.092895
set_missing_host_key_policy(): 0:00:00.000015
connect(): 0:00:01.529776

paramiko: 1.13.0
SSHClient(): 0:00:00.000027
load_system_host_keys(): 0:00:01.035239
set_missing_host_key_policy(): 0:00:00.000022
connect(): 0:00:01.463467

paramiko: 1.13.0
SSHClient(): 0:00:00.000026
load_system_host_keys(): 0:00:00.766748
set_missing_host_key_policy(): 0:00:00.000016
connect(): 0:00:01.543187

paramiko: 1.13.0
SSHClient(): 0:00:00.000026
load_system_host_keys(): 0:00:00.998344
set_missing_host_key_policy(): 0:00:00.000021
connect(): 0:00:01.484048

paramiko: 1.13.0
SSHClient(): 0:00:00.000042
load_system_host_keys(): 0:00:01.157835
set_missing_host_key_policy(): 0:00:00.000017
connect(): 0:00:01.472156

[lndbrg]

Cool, that will land in the next release that @bitprophet will do. Will close this since we already have a fix!