Schema Registry client and SSL

[dodysw]

Would it be possible to enhance the new AvroConsumer/AvroProducer/CachedSchemaRegistryClient to optionally accept certs related parameters so we can connect to Schema Registry on SSL channel that requires client certificate authentication. Or possibly accept a requests.Session() object (I think we can put a persistent certificate parameter to session object)

[ewencp]

@roopahc @criccomini Any thoughts on this? I'm a bit skeptical of exposing implementation details like the use of requests in the public interface. Is it possible to achieve something similar just by forwarding *args and **kwargs maybe?

[criccomini]

Agree on not wanting to expose requests.

[edenhill]

@dodysw Can you help us identify exactly what parameters are needed? Is certs= and verify= enough?

I would prefer to see this added as config dict properties rather than constructor args.

[dodysw]

For python requests, those are pretty much what needed. It looks like this:

ssl_cafile = '/opt/logs/sandbox/cert/kafka/ca-cert.pem'
ssl_certfile = '/opt/logs/sandbox/cert/kafka/client-cert.pem'
ssl_keyfile = '/opt/logs/sandbox/cert/kafka/client-key-plain.pem'
s = requests.Session()
s.cert = (ssl_certfile, ssl_keyfile)
s.verify = ssl_cafile

Client private key pem file must be unencrypted so it doesn't take a password.

Added:
s.cert can be just a string (path to ssl_certfile) if server doesn't require client cert authentication.
verify can be False to disable server hostname check.

[edenhill]

Okay, cool. What about the following new config properties:

• schema.registry.ssl.ca.location = str
• schema.registry.ssl.certificate.location = str
• schema.registry.ssl.key.location = str

This is in line with existing SSL config properties for Kafka.
(Search for ssl.ca.location here: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)

[dodysw]

Close, I think schema.registry.ssl.certificate.location shouldn't be a tuple if schema.registry.ssl.key.location is there.

[roopahc]

@ewencp Should be possible since we are hitting rest end points after all. We can take them as argument to AvroProducer, AvroConsumer classes

[oh54]

Commenting to say that I would very much be interested in being able to turn off server certificate verification when using Schema Registry over HTTPS.

[madtimber]

Am I correct in reading thru this, that there is no current way to use this library with a schema registry over HTTPS?

@rnpridgeon I notice you self assigned this issue to yourself. Are you currently working on adding this functionality?

[madtimber]

So I'm realizing now that my issue is different from @dodysw. I don't need to do client cert authentication, just need to verify the server's cert. And there's a way to handle that.

My solution, for anyone else searching for something similar, was to set the REQUESTS_CA_BUNDLE environment variable with our combined PEMs from our truststore.

It's a single line, easily missed, in the requests documentation:

This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.

Full docs here: http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

[rnpridgeon]

fixed with #408