Python package aiohttp>=3.9.2 break azure-keyvault-certificates package's get_certificate functionality

[gsrikant7]

Describe the bug

Bug is related to this issue in azure-keyvault-certificates package.

• Package Name: aiohttp
• Package Version: 3.9.2
• Operating System: Linux
• Python Version: 3.9.14

We are trying to get certificate details using azure-keyvault-certificates:4.7.0 azure SDK that in turn uses aiohttp:3.9.2 for async transport. We are getting this error on an azure K8s cluster. The managed identity that we are using for the pod has Key Vault Certificates Officer IAM role assignment and also has the required access policies. We are getting the following error -

Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'c88c58b4-c5ab-11ee-a5a1-26350f8b0c12'
    'User-Agent': 'azsdk-python-keyvault-certificates/4.7.0 Python/3.9.14 (Linux-5.15.138.1-4.cm2-x86_64-with-glibc2.35)'
No body was attached to the request | 20240205.5 | QueueBasedNotebookExecutor | nb-execution-large
2024-02-07 11:26:53,845 | azure.core.pipeline.policies.http_logging_policy | INFO | Request URL: '[https://dex-jpn-run-kv-ppe2.vault.azure.net/certificates/dex-nbsolution-ppe/?api-version=REDACTED'](https://dex-jpn-run-kv-ppe2.vault.azure.net/certificates/dex-nbsolution-ppe/?api-version=REDACTED%27)
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'c88c58b4-c5ab-11ee-a5a1-26350f8b0c12'
    'User-Agent': 'azsdk-python-keyvault-certificates/4.7.0 Python/3.9.14 (Linux-5.15.138.1-4.cm2-x86_64-with-glibc2.35)'
No body was attached to the request | 20240205.5 | QueueBasedNotebookExecutor | nb-execution-large
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1090, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1120, in _create_connection_transport
    await waiter
  File "/usr/lib/python3.9/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 945, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/transport/_aiohttp.py", line 263, in send
    result = await self.session.request(  # type: ignore
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/client.py", line 578, in _request
    conn = await self._connector.connect(
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 544, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 911, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 1235, in _create_direct_connection
    raise last_exc
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 1204, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/venv/lib/python3.9/site-packages/aiohttp/connector.py", line 994, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host dex-jpn-run-kv-ppe2.vault.azure.net:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')]
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 258, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 90, in main
    confidential_client_application = await create_aad_confidential_client_application(azure_credential,
  File "/src/app/notebook_executor/notebook_queue_based_executor/./main.py", line 176, in create_aad_confidential_client_application
    certificate = await certificate_client.get_certificate(aad_app_configuration.certificate_name)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/aio/_client.py", line 154, in get_certificate
    bundle = await self._client.get_certificate(
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/_generated/aio/_operations_mixin.py", line 741, in get_certificate
    return await mixin_instance.get_certificate(vault_base_url, certificate_name, certificate_version, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/azure/keyvault/certificates/_generated/v7_4/aio/operations/_key_vault_client_operations.py", line 1847, in get_certificate
    pipeline_response: PipelineResponse = await self._client._pipeline.run(  # type: ignore # pylint: disable=protected-access
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 221, in run
    return await first_node.send(pipeline_request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  [Previous line repeated 2 more times]
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect_async.py", line 73, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry_async.py", line 205, in send
    raise err
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry_async.py", line 179, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 94, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  [Previous line repeated 2 more times]
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 106, in send
    await self._sender.send(request.http_request, **request.context.options),
  File "/opt/venv/lib/python3.9/site-packages/azure/core/pipeline/transport/_aiohttp.py", line 300, in send
    raise ServiceRequestError(err, error=err) from err
azure.core.exceptions.ServiceRequestError: Cannot connect to host dex-jpn-run-kv-ppe2.vault.azure.net:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')]
Stream closed EOF for jpn/notebook-executor-largesku-8689ffbdb-przg5 (wdatp-service)

To Reproduce

Steps to reproduce the behavior -

1. Install azure-identity==1.13.0, azure-keyvault==4.2.0, azure-keyvault-certificates==4.7.0, and aiohttp==3.9.2 / 3.9.3 in a Linux machine.
2. Use the below code -

from azure.identity.aio import DefaultAzureCredential
from azure.keyvault.certificates.aio import CertificateClient

azure_credential = DefaultAzureCredential()
certificate_client = CertificateClient(vault_url=<keyvault_url>, credential=azure_credential)
certificate = await certificate_client.get_certificate(<cerificate_name>)

Expected behavior

get_certificate should return a KeyVaultCertificate object.
Instead, we get the error mentioned above.

Logs/tracebacks

Error log is already shared above

Python Version

$ python --version
3.9.14

aiohttp Version

$ python -m pip show aiohttp
3.9.2

multidict Version

$ python -m pip show multidict
6.0.4

yarl Version

$ python -m pip show yarl
1.9.2

OS

Linux

Related component

Client

Additional context

• The same code is working fine on a local machine (windows) but failing in AKS on Linux.
• We moved from using aiohttp==3.9.0 to aiohttp==3.9.2 and we see that this change in 3.9.2, fiddles with the SSL parameter.
• This change was added in a patch version which is now breaking our code. This change should have been in a major version.

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[Dreamsorcerer]

I don't see anything obviously wrong. But, the only thing that could have changed (as a result of the linked issue) is that it was expecting ssl previously, but actually not enabling ssl. Now it actually enables ssl, and you are missing certificates.

[Dreamsorcerer]

i.e. It may have been sending ssl=True somewhere, which was an invalid value before that change and actually disables ssl.

To fix your issue, install certificates so Python can find them (most installs do this by default), or use certifi as shown as an example in the docs.

[gsrikant7]

Thanks @Dreamsorcerer for the quick turnaround.

We are not directly using aiohttp. We are using it via azure-keyvault-certificates from azure python sdk. Please check this issue that we have raised with them. We have directed your query to them.

[webknjaz]

I doubt someone here would be debugging a third party project. We need a pure-aiohttp reproducer to move forward.

[Dreamsorcerer]

We are not directly using aiohttp

Neither of my suggestions require direct use of aiohttp. Installing certificates globally is a normal thing that affects all of Python. And the project appears to have some way to specify the SSL context that gets passed to aiohttp if you need to use certifi instead.