ssl: Support using client-side SSL certificates, closes #253 and #447

[Ormod]

Kafka 0.9.0+ supports using SSL encrypted connections authenticated
with client certificates. This adds support for these. After this
when using SSL/TLS for Kafka broker connections,
TLSv1.0+ is used for authentication, protocols earlier than that are
deprecated from the get go as insecure.

If keyfile/certificate are left empty, the client can still use
SSL/TLS for the transport, but in that case the server must be
configured to accept unauthenticated users.

[dpkp]

I'm going to hold off on this for now -- would prefer to include along with a larger 0.9 support release

[Ormod]

Well the previous user code would still work with 0.8 as well without any client changes since the commit is backwards compatible and SSL needs to be turned on explicitly for anything to change.

But if you do want a larger 0.9 supporting release what features would you actually like to see in it?

[dpkp]

Yes, I agree that this could be merged w/o breaking 0.8 support. But I think kafka-python needs a quick maintenance release now, and then plan to follow on with a version bump and add 0.9 feature support. I think it will be easier for everyone if we keep maintenance / bug fixes separate from the new features.

0.9 client release should include: SSL (this PR), Consumer Group APIs (in progress), Metrics (another open PR, needs a little work). Consumer Group work also includes a fairly substantial refactor of the protocol layer. Hope to get this pushed out in the next week or two.

[Ormod]

Ok, sounds fine, thanks for the info.

[davidrabinowitz]

Hi @dpkp,

Now that version 1.0 has been released, is it possible to push the 0.9 features ( and especially this one ) forward? We would be happy to assist if needed.

[dpkp]

Yes! Thanks for the ping. I have a local branch that ports your code to the
new client. Let me push that to github and we can talk about what is left
to close this out
On Feb 22, 2016 12:07 PM, "David Rabinowitz" notifications@github.com
wrote:

Hi @dpkp https://github.com/dpkp,

Now that version 1.0 has been released, is it possible to push the 0.9
features ( and especially this one ) forward? We would be happy to assist
if needed.

—
Reply to this email directly or view it on GitHub
#474 (comment).

[dpkp]

branch is at https://github.com/dpkp/kafka-python/tree/ssl_support

[Ormod]

I read through your branch and it looks ok from my POV. Any chances of getting it merged?

[dpkp]

I haven't done a lot of testing on that branch, but I am happy to merge if it seems sane.

[Ormod]

I finally tried this (and the the rest of the recent client changes) out but unfortunately couldn't make it work. One thing I did notice while trying to get the new client to work was that missing from these changes are at least config keywords from KafkaConsumer and KafkaProducer since it'll raise an error if any of the new ssl_* config keywords are defined.

The point where it fails for me is in the SSL handshake: (added a log.exception there)

Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/python3_kafka-1.0.2.dev0-py3.4.egg/kafka/conn.py", line 163, in _try_handshake
self._sock.do_handshake()
File "/usr/lib64/python3.4/ssl.py", line 810, in do_handshake
self._sslobj.do_handshake()
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:600)

After this I tried seeing if it'd work with the latest git master but the patch no longer applies cleanly. Anyway, the ssl_* config keywords should be added to the KafkaConsumer and KafkaProducer and the commit should be rebased on top of master. I'm not quite sure what's going wrong with the handshake since the certs/keys/ca's are identical to what I've been using with my original version of the SSL patch.

[dpkp]

Thanks -- I will update and push a new PR so testing can be more explicit.

[dpkp]

Moved to #621