Keep getting ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)

[dariodariodario]

Hello,
I am using kafka-python>=2.0.1 (with Python 3.7.6 on MacOsx, build from the python website) it generally works fine, but when I tried to access our broker, that has SSL auth, I will get the above error. it happens if I enable SSL with my certificates. I'm sure that there's one that is self signed internally in my company.

return KafkaProducer(
            bootstrap_servers=self.bootstrap_servers,
            acks='all',
            compression_type=None,
            retries=5,
            batch_size=16384 * 5,
            max_block_ms=5000,
            retry_backoff_ms=100 * 10,
            linger_ms=5,
            client_id='data-importer',
            security_protocol='SSL',
            ssl_check_hostname=True,
            api_version=(0, 20),
            ssl_cafile=rel_to(__file__, '../kafkakeys/KafkaClientCAChain.pem'),
            ssl_certfile=rel_to(__file__, '../kafkakeys/certificate.pem'),
            ssl_keyfile=rel_to(__file__, '../kafkakeys/key.pem'),

with this, when sending a message, it gets stuck in a loop and prints:

Traceback (most recent call last):
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 60, in run
    self.run_once()
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 160, in run_once
    self._client.poll(timeout_ms=poll_timeout_ms)
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 600, in poll
    self._poll(timeout / 1000)
  File "/Users/d-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 646, in _poll
    conn.connect()
  File "/Users/d-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
    if self._try_handshake():
  File "/Users/-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
    self._sock.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)
ERROR:kafka.producer.sender:Uncaught error in kafka producer I/O thread
Traceback (most recent call last):
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 60, in run
    self.run_once()
  File "/Users/-----dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 160, in run_once
    self._client.poll(timeout_ms=poll_timeout_ms)
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 580, in poll
    self._maybe_connect(node_id)
  File "/Users/=------/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 390, in _maybe_connect
    conn.connect()
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
    if self._try_handshake():
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
    self._sock.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

to solve this I tried a number of python installations (provided by brew, pyenv and eventually the installer from the python website). I tried to fix the issue by running Install Certificates.command that reinstalls the certificates. but it keeps doing so.

If I remove ssl_cafile and ssl_certfile (or just one of the two, leaving ssl_keyfile) it will stop giving that exception, but will just die with

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 5.0 secs.

Can anyone help please?

[jeffwidman]

This looks like an error with how you're passing in the certificate stuff. In particular, your chain may be setup incorrect, which is completely out of control of this library. If you can provide more details that proves the problem is with this library and not your config, then happy to look at this further, but until then I'm 95% sure this is a user/configuration error.

[alekskar]

Hi! @jeffwidman I faced with the same issue. We are using strimzi kafka operator for k8s which creates self-signed certificates to authenticate users over tls listener.
Originally issue was caused by value of SSLContext.verify_mode of ssl lib. Kafka python conn.py provides 2 options: CERT_OPTIONAL or CERT_REQUIRED in both cases based on SSL lib documentation :

any verification error immediately aborts TLS handshake

In case of self-signed cert we can observe this validation error with openssl tool:
openssl s_client -connect my-cluster-kafka-bootstrap:9093

..... ( output suppressed)
---
SSL handshake has read 5054 bytes and written 452 bytes
Verification error: self signed certificate in certificate chain
.....
 Verify return code: 19 (self signed certificate in certificate chain)

The same error returns python script during execution.
My proposal is to add one more parameter let's say skip_tls_verifications to enable
self._ssl_context.verify_mode = ssl.CERT_NONE for developers purposes ONLY.
Thanks!

[henriavo]

I'm getting this same error. Python 3.7.6. kafka-python==2.0.1. BUT I am running this with a local deployment of kafka. So user configuration as @jeffwidman suggested may not be the root cause. Any help is appreciated!

    producer = KafkaProducer(bootstrap_servers=['localhost:2181'],
                             security_protocol="SSL",
                             ssl_check_hostname=False)

Traceback (most recent call last):
  File "performance.py", line 24, in <module>
    ssl_check_hostname=False)
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/producer/kafka.py", line 382, in __init__
    **self.config)
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/client_async.py", line 242, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/client_async.py", line 907, in check_version
    version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 1228, in check_version
    if not self.connect_blocking(timeout_at - time.time()):
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 337, in connect_blocking
    self.connect()
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
    if self._try_handshake():
  File "/Users/hidro/workspace/turnip-adid/venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
    self._sock.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

[prannay-srivastava]

Try this:
python -m pip install azure-eventhub-checkpointstoreblob-aio --trusted-host=pypi.python.org --trusted-host=pypi.org --trusted-host=files.pythonhosted.org

[mihai-dobre]

@henriavo this is the config that i use for Producer
pip freeze:

confluent-kafka==1.5.0
pyOpenSSL==19.1.0

on Python 3.8.6

config = {
    "bootstrap_server": "ssl://127.0.0.1:9094"
    "security.protocol": "SSL",
    "ssl.ca.location": "../ssl/root/intermediate/ca-chain.cert.pem",
    "ssl.certificate.location": "../ssl/root/intermediate/producer/producer.cert.pem",
    "ssl.key.location": "../ssl/root/intermediate/producer/producer.key.pem",
    "ssl.key.password": "qwerty"}

ca-chain is intermediate-certificate + root-certificate, producer.cert is signed with the intermediate certificate. It worked for me. Also, enable the DEBUG logs on the broker to get more info on the errors you're getting. The errors helped me a lot in debugging SSL problems.
This is the configuration I use for brokers(on docker):

    image: confluentinc/cp-kafka:5.5.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    volumes:
      - ./ssl/root/intermediate/broker:/etc/kafka/secrets
    ports:
      - 9092:9092
      - 9093:9093
      - 9094:9094
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
      KAFKA_ZOOKEEPER_SSL_CLIENT_ENABLE: 'true'
      KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET: org.apache.zookeeper.ClientCnxnSocketNetty
      KAFKA_ZOOKEEPER_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
      KAFKA_ZOOKEEPER_SSL_KEYSTORE_PASSWORD: confluent
      KAFKA_ZOOKEEPER_SSL_KEYSTORE_TYPE: PKCS12
      KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/broker.truststore.jks
      KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_PASSWORD: confluent

      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,SSL:SSL,PLAINTEXT:PLAINTEXT,SSL_INTERNAL:SSL
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://localhost:9092,SSL://localhost:9094,SSL_INTERNAL://broker:9093

      KAFKA_LOG4J_LOGGERS: "kafka.authorizer.logger=INFO"
      KAFKA_LOG4J_ROOT_LOGLEVEL: INFO

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1

      KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker.keystore.jks
      KAFKA_SSL_KEYSTORE_FILENAME: broker.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_creds
      KAFKA_SSL_KEYSTORE_PASSWORD: confluent

      KAFKA_SSL_KEY_CREDENTIALS: broker_creds
      KAFKA_SSL_KEY_PASSWORD: confluent

      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/broker.truststore.jks
      KAFKA_SSL_TRUSTSTORE_FILENAME: broker.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: confluent
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_creds

      KAFKA_SSL_CLIENT_AUTH: "required"

#      KAFKA_OPTS: "-Djavax.net.debug=all"

KAFKA_OPTS is used to enable debug on the broker side. You can enable just ssl debug with something like -Djavax.net.debug=ssl(if i remember correctly)

[ImadYamane]

Hi @jeffwidman , I am facing the same issue with strimzi. How did you solve yours? Thanks

[abiodunjames]

Hello, I'm facing the same issue with strimzi right now.

[abiodunjames]

I'm currently using the sslcontext as a workaround

cert = "user.crt"
key = "user.key"
context = ssl.create_default_context()
context.load_cert_chain(certfile=cert, keyfile=key)
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
self.ssl_context = context
self.security_protocol = "SSL"

[OuesFa]

@ImadYamane @alekskar how did you please solve the issue when using Strimzi self signed certs ? And what Strimzi files did you use as ssl_cafile and ssl_certfile ?

[dechoma]

it is significant where you get your ssl_cafile from

when i was using ca.crt ( from user's secret ) as ssl_cafile then I get an error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain , when I switched to the ca.crt from secret cluster-ca-cert it worked like a charm

[rafis]

Thank you @dechoma . If anybody is using Strimzi Kafka Operator then you should take ca.crt from cluster-cluster-ca-crt and not from cluster-clients-ca-crt.

[zdforient]

for thoes suffer from the problem and use gmail account to download some files or use to generate some confidential files, try to use another email account other than *@gmail.com.

[rdrigos]

Estou obtendo o seguinte erro:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:992)

Quando tento me conectar ao broker(Mosquitto que está no docker, ou sejá, um Pseudo-Broker) onde está utilizando o SSL. Gerei minha CA, certificados e chaves para o cliente e servidor e e está devidamente configurada no mosquito.conf, e só apresenta erro na conexão usando o PAHO ou MQTTX.

Alguem poderia ajudar?

[dpkp]

Use the ssl_context configuration parameter to address this issue.