This line:
assert ecdsa.point_is_valid(curve.generator, x, y)
takes about 26ms to complete. With my ssh known hosts file has 182 ecdsa keys in, this line alone causes a paramiko client to spend almost five seconds before even connecting to to the server. As the known_hosts file is a in essence a database of trusted keys, I think it should be possible to skip this assert in this case.