SSLCertVerificationError on trying to get_certificate using aiohttp python package

[gsrikant7]

Creating a bug similar to this issue. Please find below the details -

• Package Name: azure-keyvault-certificates
• Package Version: 4.7.0
• Operating System: Linux
• Python Version: 3.9.14

Describe the bug

We are trying to get certificate details using azure-keyvault-certificates:4.7.0 azure SDK that in turn uses aiohttp:3.9.2 as async transport. We are getting this error on an azure K8s cluster. The managed identity that we are using for the pod has Key Vault Certificates Officer IAM role assignment and also has the required access policies. We are getting the following error -

Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'c88c58b4-c5ab-11ee-a5a1-26350f8b0c12'
    'User-Agent': 'azsdk-python-keyvault-certificates/4.7.0 Python/3.9.14 (Linux-5.15.138.1-4.cm2-x86_64-with-glibc2.35)'
No body was attached to the request | 20240205.5 | QueueBasedNotebookExecutor | nb-execution-large
2024-02-07 11:26:53,845 | azure.core.pipeline.policies.http_logging_policy | INFO | Request URL: '[https://dex-jpn-run-kv-ppe2.vault.azure.net/certificates/dex-nbsolution-ppe/?api-version=REDACTED'](https://dex-jpn-run-kv-ppe2.vault.azure.net/certificates/dex-nbsolution-ppe/?api-version=REDACTED%27)
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'c88c58b4-c5ab-11ee-a5a1-26350f8b0c12'
    'User-Agent': 'azsdk-python-keyvault-certificates/4.7.0 Python/3.9.14 (Linux-5.15.138.1-4.cm2-x86_64-with-glibc2.35)'
No body was attached to the request | 20240205.5 | QueueBasedNotebookExecutor | nb-execution-large
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1090, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1120, in _create_connection_transport
    await waiter
  File "/usr/lib/python3.9/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 945, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/transport/_aiohttp.py", line 263, in send
    result = await self.session.request(  # type: ignore
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/client.py", line 578, in _request
    conn = await self._connector.connect(
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 544, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 911, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 1235, in _create_direct_connection
    raise last_exc
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 1204, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 994, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host dex-jpn-run-kv-ppe2.vault.azure.net:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')]
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 258, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 90, in main
    confidential_client_application = await create_aad_confidential_client_application(azure_credential,
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 176, in create_aad_confidential_client_application
    certificate = await certificate_client.get_certificate(aad_app_configuration.certificate_name)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/aio/_client.py", line 154, in get_certificate
    bundle = await self._client.get_certificate(
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/_generated/aio/_operations_mixin.py", line 741, in get_certificate
    return await mixin_instance.get_certificate(vault_base_url, certificate_name, certificate_version, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/_generated/v7_4/aio/operations/_key_vault_client_operations.py", line 1847, in get_certificate
    pipeline_response: PipelineResponse = await self._client._pipeline.run(  # type: ignore # pylint: disable=protected-access
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 221, in run
    return await first_node.send(pipeline_request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  [Previous line repeated 2 more times]
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect_async.py", line 73, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry_async.py", line 205, in send
    raise err
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry_async.py", line 179, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 94, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  [Previous line repeated 2 more times]
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 106, in send
    await self._sender.send(request.http_request, **request.context.options),
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/transport/_aiohttp.py", line 300, in send
    raise ServiceRequestError(err, error=err) from err
azure.core.exceptions.ServiceRequestError: Cannot connect to host dex-jpn-run-kv-ppe2.vault.azure.net:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')]
Stream closed EOF for jpn/notebook-executor-largesku-8689ffbdb-przg5 (wdatp-service)

To Reproduce
Steps to reproduce the behavior -

Install azure-identity==1.13.0, azure-keyvault==4.2.0, azure-keyvault-certificates==4.7.0, and aiohttp==3.9.2 / 3.9.3

from azure.identity.aio import DefaultAzureCredential
from azure.keyvault.certificates.aio import CertificateClient

azure_credential = DefaultAzureCredential()
certificate_client = CertificateClient(vault_url=<keyvault_url>, credential=azure_credential)
certificate = await certificate_client.get_certificate(<cerificate_name>)

Expected behavior
get_certificate should return a KeyVaultCertificate object. Instead, we get the above error.

Additional Context

• The same code is working fine on a local machine (windows).
• We moved from using aiohttp==3.9.0 to aiohttp==3.9.2 and we see that this change in 3.9.2, fiddles with the SSL parameter.

[kashifkhan]

Thank you for the feedback @gsrikant7 . We will investigate and get back to you asap.

[mccoyp]

Hi @gsrikant7, thank you for the details. When you say that the same code works when running locally on Windows, are you making requests to the same Key Vault as the vault being used in AKS, dex-jpn-run-kv-ppe2? If you're communicating with the same vault in each environment, could I ask you to try using the synchronous CertificateClient instead of the async one to run this get_certificate operation?

If you still see the same behavior, it's possible that the AKS cluster needs to update its certificates. I'm not familiar enough with AKS to give advice, but I did find this documentation that could be useful: https://learn.microsoft.com/azure/aks/certificate-rotation

[github-actions]

Hi @gsrikant7. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[gsrikant7]

@mccoyp Thanks for the quick turnaround. Yes, we are making calls to the same KV (same as the one used in AKS).

We will test it with the synchronous CertificateClient to run get_certificate and see if it works. My understanding is that it will work with the synchronous client. This is because we have tested with an older version of aiohttp==3.9.0 (the one we were previously using) and it works correctly in local as well as in AKS and was able to connect to the KV.

Can you please also let us know what is the value for ssl parameter and verify parameter that is being passed to aiohttp from azure-core ?
This is based on this comment in this issue that we have raised.

This will help in understanding SSL verification prior to aiohttp==3.9.2.

[mccoyp]

Hi @gsrikant7, thank you for your response -- please do update us when you try the synchronous client. My suspicion is that the AKS cluster has a client-side certificate issue that was being hidden by aiohttp 3.9.0.

The value of ssl in the request depends on your local version of azure-core, as well as your transport configuration. As of azure-core 1.29.5, we use the SSL context from aiohttp by default. Upgrading your versions of azure-core (to >=1.29.5) and aiohttp (to >=3.9.2) should give you the most accurate picture of your AKS environment's certificate setup.

The synchronous client will be unaffected by the state of aiohttp, which is why using it should also help us identify the issue.

[github-actions]

Hi @gsrikant7. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @gsrikant7, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[gsrikant7]

Hi @mccoyp. Apologies for the delay in reply.

We have now tried the synchronous CertificateClient from azure-keyvault-certificates:4.7.0. It worked perfectly fine in local (Windows) and started the service. When we deployed the change in AKS cluster - we found out that it was able to get the certificate and was able to create the confidential application (which was failing earlier in async version) - but it failed at another AIO call - BlobServiceClient.

This means that the issue is not with azure resource SDKs that we are using to connect to specific azure resources - but with azure-core:1.29.4 as azure-core is responsible for providing the transport (aiohttp in this case) that is used to connect to azure resources.

So, we tried fixing the version of azure-core to 1.30.0 (the latest released version) and tested it with aiohttp:3.9.2. Still no luck and it continues to fail with same error.

This leads us to believe that azure-core must check the usage of aiohttp for transport and check. In parallel, we would like to understand if we can use any other async transport (other than aiohttp) and if there is any wiki for it that we can use.

[xiangyan99]

@gsrikant7 Do you happen to remember the azure-core version when you moved from using aiohttp==3.9.0 to aiohttp==3.9.2 and the issue started to happen?

[github-actions]

Hi @gsrikant7. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[gsrikant7]

@xiangyan99 When we moved to use aiohttp:3.9.2 - we were using azure-core:1.29.4.

But we also tested it with azure-core:1.30.0 and aiohttp:3.9.2. In both cases it is failing with same error.

[mccoyp]

Hi @gsrikant7, thank you for the information. It would help to determine which package is introducing the issue by trying each combination of "old" and "new" versions for azure-core and aiohttp. Could I ask you to try installing the following versions of each library and reporting the behavior you see from an async CertificateClient in each of these four configurations?

• azure-core==1.29.4 and aiohttp==3.9.0
• azure-core==1.30.0 and aiohttp==3.9.0
• azure-core==1.29.4 and aiohttp==3.9.3
• azure-core==1.30.0 and aiohttp==3.9.3

[github-actions]

Hi @gsrikant7. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[gsrikant7]

@mccoyp Please find below the various configurations and their behavior -

azure-core==1.29.4 and aiohttp==3.9.0 -- Old Set Up - Works fine
azure-core==1.30.0 and aiohttp==3.9.0 -- Works fine
azure-core==1.29.4 and aiohttp==3.9.3 -- New Set Up - Does not work fine (SSLCertVerificationError)
azure-core==1.30.0 and aiohttp==3.9.3 -- Does not work fine (SSLCertVerificationError)

Please let me know if you need more information.

[mccoyp]

Hi @gsrikant7, thank you for confirming the behavior! Since upgrading aiohttp exposes the failure independently of your azure-core version, it looks like aiohttp is somehow causing the problem. I would suggest trying a different async transport, like trio (which you may need to pip install).

The Transport section of Core's client library developer documentation shows some different async transport options. To use trio, you can import the TrioRequestsTransport from azure.core.pipeline.transport and provide it to both your credential and client with the transport keyword argument. For example:

from azure.identity.aio import DefaultAzureCredential
from azure.keyvault.certificates.aio import CertificateClient
from azure.core.pipeline.transport import TrioRequestsTransport

credential = DefaultAzureCredential(transport=TrioRequestsTransport())
client = CertificateClient(VAULT_URL, credential, transport=TrioRequestsTransport())

[github-actions]

Hi @gsrikant7. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @gsrikant7, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[gsrikant7]

Hi @mccoyp Apologies for the delay - we were busy with other items.

I tried making these changes and found out that the changes are not trivial. Plus trio and asyncio cannot be used right away, without some sort of a shim to move from trio to asyncio boundary and vice-versa.

The use of a shim will also add performance overhead than the current system.

Can you please let us know how to proceed here ?

[gsrikant7]

@mccoyp Can we also have someone from azure-core team as well ?

[mccoyp]

Hi @gsrikant7, I understand that switching your system over to a different transport could take a good amount of work (and a full migration would be necessary, since mixing trio and aiohttp isn't recommended). After discussing your scenario with our folks from Core, it sounds like pinning your aiohttp dependency to version 3.9.0 might be best, at least in the short-term.

Is there any reason that pinning your aiohttp dependency would be an issue? It seems from #34244 (comment) that aiohttp is the source of the problem.

[slenin]

Hi, I face the same issue working with Azure key vault (aio) from python.
It started happening from azure-core==1.30.0.
azure-core==1.29.4 worked fine.

[mccoyp]

Hi @slenin, do you know which versions of aiohttp you were using in each case (azure-core 1.29.4 vs 1.30.0)? We found that upgrading aiohttp was causing @gsrikant7's issue, rather than azure-core.

[slenin]

Hi @slenin, do you know which versions of aiohttp you were using in each case (azure-core 1.29.4 vs 1.30.0)? We found that upgrading aiohttp was causing @gsrikant7's issue, rather than azure-core.

The aiohttp version is 3.9.0.

azure-core==1.29.4, aiohttp==3.9.0 - Works
azure-core==1.29.5, aiohttp==3.9.0 - Raises the exception (SSLCertVerificationError)
azure-core==1.30.0, aiohttp==3.9.0 - Raises the exception (SSLCertVerificationError)

[xiangyan99]

@slenin Thanks for reporting.

It seems to me the issue you faced was different from the original one.

In the original one:
azure-core==1.30.0 and aiohttp==3.9.0 -- Works fine

The difference between azure-core 1.29.4 & 1.29.5 is we enable ssl cert validation on aiohttp client since 1.29.5.

And it seems to me you might not have a valid cert installed hence it failed to connect.

To validate it, could you try sync version and see if it works?

[xiangyan99]

And we already took the original one offline and I am closing the issue.

@slenin please feel free to open a new one for your case and link to this one.